2023-03-31 16:53:33,276 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/7dffd501-dbfb-c6fc-19df-043ab2e220e3 2023-03-31 16:53:33,292 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.TestAcidGuaranteesWithEagerPolicy timeout: 13 mins 2023-03-31 16:53:33,309 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-03-31 16:53:33,310 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/7dffd501-dbfb-c6fc-19df-043ab2e220e3/cluster_ab6d4253-9335-03d0-adb0-24d05d2df50a, deleteOnExit=true 2023-03-31 16:53:33,310 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-03-31 16:53:33,311 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/7dffd501-dbfb-c6fc-19df-043ab2e220e3/test.cache.data in system properties and HBase conf 2023-03-31 16:53:33,311 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/7dffd501-dbfb-c6fc-19df-043ab2e220e3/hadoop.tmp.dir in system properties and HBase conf 2023-03-31 16:53:33,311 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/7dffd501-dbfb-c6fc-19df-043ab2e220e3/hadoop.log.dir in system properties and HBase conf 2023-03-31 16:53:33,312 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/7dffd501-dbfb-c6fc-19df-043ab2e220e3/mapreduce.cluster.local.dir in system properties and HBase conf 2023-03-31 16:53:33,313 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/7dffd501-dbfb-c6fc-19df-043ab2e220e3/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-03-31 16:53:33,313 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-03-31 16:53:33,428 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-03-31 16:53:33,852 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-03-31 16:53:33,857 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/7dffd501-dbfb-c6fc-19df-043ab2e220e3/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-03-31 16:53:33,858 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/7dffd501-dbfb-c6fc-19df-043ab2e220e3/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-03-31 16:53:33,858 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/7dffd501-dbfb-c6fc-19df-043ab2e220e3/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-03-31 16:53:33,859 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/7dffd501-dbfb-c6fc-19df-043ab2e220e3/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-03-31 16:53:33,859 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/7dffd501-dbfb-c6fc-19df-043ab2e220e3/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-03-31 16:53:33,860 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/7dffd501-dbfb-c6fc-19df-043ab2e220e3/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-03-31 16:53:33,860 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/7dffd501-dbfb-c6fc-19df-043ab2e220e3/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-03-31 16:53:33,861 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/7dffd501-dbfb-c6fc-19df-043ab2e220e3/dfs.journalnode.edits.dir in system properties and HBase conf 2023-03-31 16:53:33,861 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/7dffd501-dbfb-c6fc-19df-043ab2e220e3/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-03-31 16:53:33,862 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/7dffd501-dbfb-c6fc-19df-043ab2e220e3/nfs.dump.dir in system properties and HBase conf 2023-03-31 16:53:33,862 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/7dffd501-dbfb-c6fc-19df-043ab2e220e3/java.io.tmpdir in system properties and HBase conf 2023-03-31 16:53:33,862 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/7dffd501-dbfb-c6fc-19df-043ab2e220e3/dfs.journalnode.edits.dir in system properties and HBase conf 2023-03-31 16:53:33,863 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/7dffd501-dbfb-c6fc-19df-043ab2e220e3/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-03-31 16:53:33,863 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/7dffd501-dbfb-c6fc-19df-043ab2e220e3/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-03-31 16:53:34,482 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-03-31 16:53:34,487 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-03-31 16:53:34,809 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-03-31 16:53:35,015 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-03-31 16:53:35,034 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-03-31 16:53:35,076 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-03-31 16:53:35,112 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/7dffd501-dbfb-c6fc-19df-043ab2e220e3/java.io.tmpdir/Jetty_localhost_44087_hdfs____uxxheh/webapp 2023-03-31 16:53:35,269 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:44087 2023-03-31 16:53:35,284 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-03-31 16:53:35,284 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-03-31 16:53:35,840 WARN [Listener at localhost/43983] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-03-31 16:53:35,912 WARN [Listener at localhost/43983] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-03-31 16:53:35,936 WARN [Listener at localhost/43983] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-03-31 16:53:35,944 INFO [Listener at localhost/43983] log.Slf4jLog(67): jetty-6.1.26 2023-03-31 16:53:35,951 INFO [Listener at localhost/43983] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/7dffd501-dbfb-c6fc-19df-043ab2e220e3/java.io.tmpdir/Jetty_localhost_33755_datanode____2ut34u/webapp 2023-03-31 16:53:36,095 INFO [Listener at localhost/43983] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:33755 2023-03-31 16:53:36,443 WARN [Listener at localhost/44135] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-03-31 16:53:36,979 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xe6044393fc5b69ac: Processing first storage report for DS-61656650-3fed-4084-8721-5e6c19a216a0 from datanode aeea1192-6f68-4306-8c37-723f39ed30f4 2023-03-31 16:53:36,981 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xe6044393fc5b69ac: from storage DS-61656650-3fed-4084-8721-5e6c19a216a0 node DatanodeRegistration(127.0.0.1:40953, datanodeUuid=aeea1192-6f68-4306-8c37-723f39ed30f4, infoPort=43665, infoSecurePort=0, ipcPort=44135, storageInfo=lv=-57;cid=testClusterID;nsid=1533437151;c=1680281614560), blocks: 0, hasStaleStorage: true, processing time: 2 msecs, invalidatedBlocks: 0 2023-03-31 16:53:36,982 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xe6044393fc5b69ac: Processing first storage report for DS-fdadf245-424a-4978-814d-365b044c58e5 from datanode aeea1192-6f68-4306-8c37-723f39ed30f4 2023-03-31 16:53:36,982 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xe6044393fc5b69ac: from storage DS-fdadf245-424a-4978-814d-365b044c58e5 node DatanodeRegistration(127.0.0.1:40953, datanodeUuid=aeea1192-6f68-4306-8c37-723f39ed30f4, infoPort=43665, infoSecurePort=0, ipcPort=44135, storageInfo=lv=-57;cid=testClusterID;nsid=1533437151;c=1680281614560), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-03-31 16:53:37,030 DEBUG [Listener at localhost/44135] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/7dffd501-dbfb-c6fc-19df-043ab2e220e3 2023-03-31 16:53:37,111 INFO [Listener at localhost/44135] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/7dffd501-dbfb-c6fc-19df-043ab2e220e3/cluster_ab6d4253-9335-03d0-adb0-24d05d2df50a/zookeeper_0, clientPort=51916, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/7dffd501-dbfb-c6fc-19df-043ab2e220e3/cluster_ab6d4253-9335-03d0-adb0-24d05d2df50a/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/7dffd501-dbfb-c6fc-19df-043ab2e220e3/cluster_ab6d4253-9335-03d0-adb0-24d05d2df50a/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-03-31 16:53:37,125 INFO [Listener at localhost/44135] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=51916 2023-03-31 16:53:37,136 INFO [Listener at localhost/44135] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-31 16:53:37,139 INFO [Listener at localhost/44135] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-31 16:53:37,800 INFO [Listener at localhost/44135] util.FSUtils(471): Created version file at hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7 with version=8 2023-03-31 16:53:37,800 INFO [Listener at localhost/44135] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/hbase-staging 2023-03-31 16:53:38,136 INFO [Listener at localhost/44135] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-03-31 16:53:38,654 INFO [Listener at localhost/44135] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-03-31 16:53:38,687 INFO [Listener at localhost/44135] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-03-31 16:53:38,687 INFO [Listener at localhost/44135] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-03-31 16:53:38,688 INFO [Listener at localhost/44135] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-03-31 16:53:38,688 INFO [Listener at localhost/44135] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-03-31 16:53:38,688 INFO [Listener at localhost/44135] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-03-31 16:53:38,843 INFO [Listener at localhost/44135] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-03-31 16:53:38,915 DEBUG [Listener at localhost/44135] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-03-31 16:53:39,008 INFO [Listener at localhost/44135] ipc.NettyRpcServer(120): Bind to /172.31.14.131:43681 2023-03-31 16:53:39,018 INFO [Listener at localhost/44135] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-31 16:53:39,020 INFO [Listener at localhost/44135] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-31 16:53:39,040 INFO [Listener at localhost/44135] zookeeper.RecoverableZooKeeper(93): Process identifier=master:43681 connecting to ZooKeeper ensemble=127.0.0.1:51916 2023-03-31 16:53:39,179 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:436810x0, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-03-31 16:53:39,182 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:43681-0x1013b7eac850000 connected 2023-03-31 16:53:39,278 DEBUG [Listener at localhost/44135] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-03-31 16:53:39,279 DEBUG [Listener at localhost/44135] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-03-31 16:53:39,283 DEBUG [Listener at localhost/44135] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-03-31 16:53:39,292 DEBUG [Listener at localhost/44135] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=43681 2023-03-31 16:53:39,296 DEBUG [Listener at localhost/44135] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=43681 2023-03-31 16:53:39,296 DEBUG [Listener at localhost/44135] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=43681 2023-03-31 16:53:39,297 DEBUG [Listener at localhost/44135] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=43681 2023-03-31 16:53:39,297 DEBUG [Listener at localhost/44135] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=43681 2023-03-31 16:53:39,303 INFO [Listener at localhost/44135] master.HMaster(444): hbase.rootdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7, hbase.cluster.distributed=false 2023-03-31 16:53:39,375 INFO [Listener at localhost/44135] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-03-31 16:53:39,376 INFO [Listener at localhost/44135] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-03-31 16:53:39,376 INFO [Listener at localhost/44135] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-03-31 16:53:39,376 INFO [Listener at localhost/44135] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-03-31 16:53:39,376 INFO [Listener at localhost/44135] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-03-31 16:53:39,376 INFO [Listener at localhost/44135] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-03-31 16:53:39,381 INFO [Listener at localhost/44135] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-03-31 16:53:39,384 INFO [Listener at localhost/44135] ipc.NettyRpcServer(120): Bind to /172.31.14.131:33441 2023-03-31 16:53:39,387 INFO [Listener at localhost/44135] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-03-31 16:53:39,395 DEBUG [Listener at localhost/44135] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-03-31 16:53:39,396 INFO [Listener at localhost/44135] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-31 16:53:39,399 INFO [Listener at localhost/44135] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-31 16:53:39,401 INFO [Listener at localhost/44135] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:33441 connecting to ZooKeeper ensemble=127.0.0.1:51916 2023-03-31 16:53:39,405 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:334410x0, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-03-31 16:53:39,406 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:33441-0x1013b7eac850001 connected 2023-03-31 16:53:39,407 DEBUG [Listener at localhost/44135] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-03-31 16:53:39,408 DEBUG [Listener at localhost/44135] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-03-31 16:53:39,409 DEBUG [Listener at localhost/44135] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-03-31 16:53:39,410 DEBUG [Listener at localhost/44135] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=33441 2023-03-31 16:53:39,410 DEBUG [Listener at localhost/44135] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=33441 2023-03-31 16:53:39,410 DEBUG [Listener at localhost/44135] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=33441 2023-03-31 16:53:39,411 DEBUG [Listener at localhost/44135] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=33441 2023-03-31 16:53:39,411 DEBUG [Listener at localhost/44135] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=33441 2023-03-31 16:53:39,413 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,43681,1680281617954 2023-03-31 16:53:39,422 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-03-31 16:53:39,424 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,43681,1680281617954 2023-03-31 16:53:39,445 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-03-31 16:53:39,445 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-03-31 16:53:39,445 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-31 16:53:39,446 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-03-31 16:53:39,447 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,43681,1680281617954 from backup master directory 2023-03-31 16:53:39,447 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-03-31 16:53:39,450 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,43681,1680281617954 2023-03-31 16:53:39,450 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-03-31 16:53:39,451 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-03-31 16:53:39,451 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase4.apache.org,43681,1680281617954 2023-03-31 16:53:39,455 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-03-31 16:53:39,457 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-03-31 16:53:39,547 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/hbase.id with ID: 05133980-5d59-4f20-a6e5-a677f484c040 2023-03-31 16:53:39,595 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-31 16:53:39,612 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-31 16:53:39,652 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x7c909e1b to 127.0.0.1:51916 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 16:53:39,684 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@307a0e54, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 16:53:39,708 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-31 16:53:39,710 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-03-31 16:53:39,727 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-03-31 16:53:39,727 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-03-31 16:53:39,729 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-03-31 16:53:39,733 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-03-31 16:53:39,734 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-03-31 16:53:39,767 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/MasterData/data/master/store-tmp 2023-03-31 16:53:39,795 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-31 16:53:39,795 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-03-31 16:53:39,795 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-31 16:53:39,795 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-31 16:53:39,795 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-03-31 16:53:39,796 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-31 16:53:39,796 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-31 16:53:39,796 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-03-31 16:53:39,797 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/MasterData/WALs/jenkins-hbase4.apache.org,43681,1680281617954 2023-03-31 16:53:39,821 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C43681%2C1680281617954, suffix=, logDir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/MasterData/WALs/jenkins-hbase4.apache.org,43681,1680281617954, archiveDir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/MasterData/oldWALs, maxLogs=10 2023-03-31 16:53:39,876 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:40953,DS-61656650-3fed-4084-8721-5e6c19a216a0,DISK] 2023-03-31 16:53:39,885 DEBUG [RS-EventLoopGroup-3-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-31 16:53:39,959 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/MasterData/WALs/jenkins-hbase4.apache.org,43681,1680281617954/jenkins-hbase4.apache.org%2C43681%2C1680281617954.1680281619831 2023-03-31 16:53:39,960 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:40953,DS-61656650-3fed-4084-8721-5e6c19a216a0,DISK]] 2023-03-31 16:53:39,961 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-03-31 16:53:39,961 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-31 16:53:39,965 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-03-31 16:53:39,967 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-03-31 16:53:40,038 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-03-31 16:53:40,045 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-03-31 16:53:40,068 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-03-31 16:53:40,083 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-31 16:53:40,090 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-03-31 16:53:40,092 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-03-31 16:53:40,111 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-03-31 16:53:40,118 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-31 16:53:40,120 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62318816, jitterRate=-0.07137727737426758}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-31 16:53:40,120 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-03-31 16:53:40,122 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-03-31 16:53:40,143 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-03-31 16:53:40,143 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-03-31 16:53:40,145 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-03-31 16:53:40,147 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-03-31 16:53:40,179 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 31 msec 2023-03-31 16:53:40,180 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-03-31 16:53:40,206 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-03-31 16:53:40,212 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-03-31 16:53:40,235 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-03-31 16:53:40,239 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-03-31 16:53:40,241 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-03-31 16:53:40,246 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-03-31 16:53:40,250 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-03-31 16:53:40,254 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-31 16:53:40,256 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-03-31 16:53:40,256 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-03-31 16:53:40,272 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-03-31 16:53:40,277 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-03-31 16:53:40,278 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-03-31 16:53:40,278 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-31 16:53:40,278 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase4.apache.org,43681,1680281617954, sessionid=0x1013b7eac850000, setting cluster-up flag (Was=false) 2023-03-31 16:53:40,300 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-31 16:53:40,308 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-03-31 16:53:40,310 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,43681,1680281617954 2023-03-31 16:53:40,317 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-31 16:53:40,324 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-03-31 16:53:40,326 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,43681,1680281617954 2023-03-31 16:53:40,329 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.hbase-snapshot/.tmp 2023-03-31 16:53:40,415 INFO [RS:0;jenkins-hbase4:33441] regionserver.HRegionServer(952): ClusterId : 05133980-5d59-4f20-a6e5-a677f484c040 2023-03-31 16:53:40,421 DEBUG [RS:0;jenkins-hbase4:33441] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-03-31 16:53:40,433 DEBUG [RS:0;jenkins-hbase4:33441] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-03-31 16:53:40,433 DEBUG [RS:0;jenkins-hbase4:33441] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-03-31 16:53:40,442 DEBUG [RS:0;jenkins-hbase4:33441] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-03-31 16:53:40,444 DEBUG [RS:0;jenkins-hbase4:33441] zookeeper.ReadOnlyZKClient(139): Connect 0x7a67ad5b to 127.0.0.1:51916 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 16:53:40,460 DEBUG [RS:0;jenkins-hbase4:33441] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@74a05c67, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 16:53:40,463 DEBUG [RS:0;jenkins-hbase4:33441] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@709fc62b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-03-31 16:53:40,463 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-03-31 16:53:40,476 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-03-31 16:53:40,477 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-03-31 16:53:40,477 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-03-31 16:53:40,477 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-03-31 16:53:40,477 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-03-31 16:53:40,477 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-31 16:53:40,477 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-03-31 16:53:40,477 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-31 16:53:40,494 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1680281650494 2023-03-31 16:53:40,497 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-03-31 16:53:40,499 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-03-31 16:53:40,499 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-03-31 16:53:40,503 DEBUG [RS:0;jenkins-hbase4:33441] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:33441 2023-03-31 16:53:40,505 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-03-31 16:53:40,507 INFO [RS:0;jenkins-hbase4:33441] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-03-31 16:53:40,507 INFO [RS:0;jenkins-hbase4:33441] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-03-31 16:53:40,507 DEBUG [RS:0;jenkins-hbase4:33441] regionserver.HRegionServer(1023): About to register with Master. 2023-03-31 16:53:40,508 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-03-31 16:53:40,511 INFO [RS:0;jenkins-hbase4:33441] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,43681,1680281617954 with isa=jenkins-hbase4.apache.org/172.31.14.131:33441, startcode=1680281619375 2023-03-31 16:53:40,517 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-03-31 16:53:40,518 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-03-31 16:53:40,518 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-03-31 16:53:40,518 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-03-31 16:53:40,519 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-03-31 16:53:40,521 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-03-31 16:53:40,522 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-03-31 16:53:40,523 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-03-31 16:53:40,528 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-03-31 16:53:40,529 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-03-31 16:53:40,532 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1680281620530,5,FailOnTimeoutGroup] 2023-03-31 16:53:40,535 DEBUG [RS:0;jenkins-hbase4:33441] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-03-31 16:53:40,536 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1680281620532,5,FailOnTimeoutGroup] 2023-03-31 16:53:40,536 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-03-31 16:53:40,536 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-03-31 16:53:40,538 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-03-31 16:53:40,538 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-03-31 16:53:40,586 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39799, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-03-31 16:53:40,599 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:40,615 DEBUG [RS:0;jenkins-hbase4:33441] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7 2023-03-31 16:53:40,615 DEBUG [RS:0;jenkins-hbase4:33441] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:43983 2023-03-31 16:53:40,615 DEBUG [RS:0;jenkins-hbase4:33441] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-03-31 16:53:40,620 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-03-31 16:53:40,621 DEBUG [RS:0;jenkins-hbase4:33441] zookeeper.ZKUtil(162): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:40,621 WARN [RS:0;jenkins-hbase4:33441] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-03-31 16:53:40,622 INFO [RS:0;jenkins-hbase4:33441] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-03-31 16:53:40,622 DEBUG [RS:0;jenkins-hbase4:33441] regionserver.HRegionServer(1947): logDir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/WALs/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:40,623 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,33441,1680281619375] 2023-03-31 16:53:40,632 DEBUG [RS:0;jenkins-hbase4:33441] zookeeper.ZKUtil(162): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:40,641 DEBUG [RS:0;jenkins-hbase4:33441] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-03-31 16:53:40,650 INFO [RS:0;jenkins-hbase4:33441] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-03-31 16:53:40,669 INFO [RS:0;jenkins-hbase4:33441] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-03-31 16:53:40,673 INFO [RS:0;jenkins-hbase4:33441] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-03-31 16:53:40,673 INFO [RS:0;jenkins-hbase4:33441] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-03-31 16:53:40,675 INFO [RS:0;jenkins-hbase4:33441] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-03-31 16:53:40,682 INFO [RS:0;jenkins-hbase4:33441] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-03-31 16:53:40,683 DEBUG [RS:0;jenkins-hbase4:33441] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-31 16:53:40,683 DEBUG [RS:0;jenkins-hbase4:33441] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-31 16:53:40,683 DEBUG [RS:0;jenkins-hbase4:33441] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-31 16:53:40,683 DEBUG [RS:0;jenkins-hbase4:33441] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-31 16:53:40,683 DEBUG [RS:0;jenkins-hbase4:33441] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-31 16:53:40,683 DEBUG [RS:0;jenkins-hbase4:33441] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-03-31 16:53:40,684 DEBUG [RS:0;jenkins-hbase4:33441] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-31 16:53:40,684 DEBUG [RS:0;jenkins-hbase4:33441] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-31 16:53:40,684 DEBUG [RS:0;jenkins-hbase4:33441] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-31 16:53:40,684 DEBUG [RS:0;jenkins-hbase4:33441] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-31 16:53:40,685 INFO [RS:0;jenkins-hbase4:33441] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-03-31 16:53:40,685 INFO [RS:0;jenkins-hbase4:33441] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-03-31 16:53:40,685 INFO [RS:0;jenkins-hbase4:33441] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-03-31 16:53:40,702 INFO [RS:0;jenkins-hbase4:33441] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-03-31 16:53:40,704 INFO [RS:0;jenkins-hbase4:33441] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,33441,1680281619375-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-03-31 16:53:40,721 INFO [RS:0;jenkins-hbase4:33441] regionserver.Replication(203): jenkins-hbase4.apache.org,33441,1680281619375 started 2023-03-31 16:53:40,721 INFO [RS:0;jenkins-hbase4:33441] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,33441,1680281619375, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:33441, sessionid=0x1013b7eac850001 2023-03-31 16:53:40,722 DEBUG [RS:0;jenkins-hbase4:33441] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-03-31 16:53:40,722 DEBUG [RS:0;jenkins-hbase4:33441] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:40,722 DEBUG [RS:0;jenkins-hbase4:33441] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,33441,1680281619375' 2023-03-31 16:53:40,722 DEBUG [RS:0;jenkins-hbase4:33441] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:40,723 DEBUG [RS:0;jenkins-hbase4:33441] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:40,723 DEBUG [RS:0;jenkins-hbase4:33441] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-03-31 16:53:40,723 DEBUG [RS:0;jenkins-hbase4:33441] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-03-31 16:53:40,723 DEBUG [RS:0;jenkins-hbase4:33441] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:40,723 DEBUG [RS:0;jenkins-hbase4:33441] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,33441,1680281619375' 2023-03-31 16:53:40,723 DEBUG [RS:0;jenkins-hbase4:33441] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-03-31 16:53:40,724 DEBUG [RS:0;jenkins-hbase4:33441] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-03-31 16:53:40,724 DEBUG [RS:0;jenkins-hbase4:33441] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-03-31 16:53:40,724 INFO [RS:0;jenkins-hbase4:33441] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-03-31 16:53:40,724 INFO [RS:0;jenkins-hbase4:33441] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-03-31 16:53:40,836 INFO [RS:0;jenkins-hbase4:33441] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C33441%2C1680281619375, suffix=, logDir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/WALs/jenkins-hbase4.apache.org,33441,1680281619375, archiveDir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/oldWALs, maxLogs=32 2023-03-31 16:53:40,854 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:40953,DS-61656650-3fed-4084-8721-5e6c19a216a0,DISK] 2023-03-31 16:53:40,857 INFO [RS:0;jenkins-hbase4:33441] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/WALs/jenkins-hbase4.apache.org,33441,1680281619375/jenkins-hbase4.apache.org%2C33441%2C1680281619375.1680281620838 2023-03-31 16:53:40,858 DEBUG [RS:0;jenkins-hbase4:33441] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:40953,DS-61656650-3fed-4084-8721-5e6c19a216a0,DISK]] 2023-03-31 16:53:40,953 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-03-31 16:53:40,954 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-03-31 16:53:40,954 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7 2023-03-31 16:53:41,371 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-31 16:53:41,374 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-03-31 16:53:41,378 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/hbase/meta/1588230740/info 2023-03-31 16:53:41,379 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-03-31 16:53:41,380 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-31 16:53:41,380 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-03-31 16:53:41,384 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/hbase/meta/1588230740/rep_barrier 2023-03-31 16:53:41,385 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-03-31 16:53:41,386 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-31 16:53:41,386 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-03-31 16:53:41,388 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/hbase/meta/1588230740/table 2023-03-31 16:53:41,389 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-03-31 16:53:41,390 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-31 16:53:41,392 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/hbase/meta/1588230740 2023-03-31 16:53:41,393 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/hbase/meta/1588230740 2023-03-31 16:53:41,397 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-31 16:53:41,399 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-03-31 16:53:41,404 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-31 16:53:41,405 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74364633, jitterRate=0.10811938345432281}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-31 16:53:41,405 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-03-31 16:53:41,405 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-03-31 16:53:41,405 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-03-31 16:53:41,406 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-03-31 16:53:41,406 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-03-31 16:53:41,406 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-03-31 16:53:41,406 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-03-31 16:53:41,407 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-03-31 16:53:41,411 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-03-31 16:53:41,411 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-03-31 16:53:41,420 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-03-31 16:53:41,435 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-03-31 16:53:41,438 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-03-31 16:53:41,590 DEBUG [jenkins-hbase4:43681] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-03-31 16:53:41,594 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,33441,1680281619375, state=OPENING 2023-03-31 16:53:41,602 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-03-31 16:53:41,605 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-31 16:53:41,605 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-03-31 16:53:41,608 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,33441,1680281619375}] 2023-03-31 16:53:41,791 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:41,793 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-03-31 16:53:41,797 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45890, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-03-31 16:53:41,808 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-03-31 16:53:41,808 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-03-31 16:53:41,812 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C33441%2C1680281619375.meta, suffix=.meta, logDir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/WALs/jenkins-hbase4.apache.org,33441,1680281619375, archiveDir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/oldWALs, maxLogs=32 2023-03-31 16:53:41,827 DEBUG [RS-EventLoopGroup-3-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:40953,DS-61656650-3fed-4084-8721-5e6c19a216a0,DISK] 2023-03-31 16:53:41,831 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/WALs/jenkins-hbase4.apache.org,33441,1680281619375/jenkins-hbase4.apache.org%2C33441%2C1680281619375.meta.1680281621813.meta 2023-03-31 16:53:41,832 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:40953,DS-61656650-3fed-4084-8721-5e6c19a216a0,DISK]] 2023-03-31 16:53:41,832 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-03-31 16:53:41,834 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-03-31 16:53:41,850 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-03-31 16:53:41,855 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-03-31 16:53:41,860 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-03-31 16:53:41,860 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-31 16:53:41,860 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-03-31 16:53:41,860 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-03-31 16:53:41,863 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-03-31 16:53:41,865 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/hbase/meta/1588230740/info 2023-03-31 16:53:41,865 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/hbase/meta/1588230740/info 2023-03-31 16:53:41,865 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-03-31 16:53:41,866 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-31 16:53:41,866 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-03-31 16:53:41,868 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/hbase/meta/1588230740/rep_barrier 2023-03-31 16:53:41,868 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/hbase/meta/1588230740/rep_barrier 2023-03-31 16:53:41,869 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-03-31 16:53:41,869 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-31 16:53:41,870 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-03-31 16:53:41,871 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/hbase/meta/1588230740/table 2023-03-31 16:53:41,871 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/hbase/meta/1588230740/table 2023-03-31 16:53:41,872 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-03-31 16:53:41,872 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-31 16:53:41,874 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/hbase/meta/1588230740 2023-03-31 16:53:41,877 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/hbase/meta/1588230740 2023-03-31 16:53:41,880 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-31 16:53:41,882 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-03-31 16:53:41,884 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69853267, jitterRate=0.04089479148387909}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-31 16:53:41,884 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-03-31 16:53:41,893 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1680281621784 2023-03-31 16:53:41,910 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-03-31 16:53:41,911 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-03-31 16:53:41,914 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,33441,1680281619375, state=OPEN 2023-03-31 16:53:41,917 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-03-31 16:53:41,917 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-03-31 16:53:41,921 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-03-31 16:53:41,921 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,33441,1680281619375 in 309 msec 2023-03-31 16:53:41,926 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-03-31 16:53:41,926 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 502 msec 2023-03-31 16:53:41,930 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 1.5560 sec 2023-03-31 16:53:41,931 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1680281621930, completionTime=-1 2023-03-31 16:53:41,931 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-03-31 16:53:41,931 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-03-31 16:53:41,991 DEBUG [hconnection-0x56ede43c-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 16:53:41,994 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45902, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 16:53:42,010 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-03-31 16:53:42,010 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1680281682010 2023-03-31 16:53:42,010 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1680281742010 2023-03-31 16:53:42,010 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 79 msec 2023-03-31 16:53:42,035 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,43681,1680281617954-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-03-31 16:53:42,035 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,43681,1680281617954-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-03-31 16:53:42,035 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,43681,1680281617954-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-03-31 16:53:42,037 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:43681, period=300000, unit=MILLISECONDS is enabled. 2023-03-31 16:53:42,037 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-03-31 16:53:42,042 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-03-31 16:53:42,053 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-03-31 16:53:42,054 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-03-31 16:53:42,063 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-03-31 16:53:42,066 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-03-31 16:53:42,068 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-31 16:53:42,088 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/hbase/namespace/4920086c0c3253e0dccccde39038f1cc 2023-03-31 16:53:42,090 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/hbase/namespace/4920086c0c3253e0dccccde39038f1cc empty. 2023-03-31 16:53:42,091 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/hbase/namespace/4920086c0c3253e0dccccde39038f1cc 2023-03-31 16:53:42,091 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-03-31 16:53:42,520 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-03-31 16:53:42,523 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => 4920086c0c3253e0dccccde39038f1cc, NAME => 'hbase:namespace,,1680281622054.4920086c0c3253e0dccccde39038f1cc.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp 2023-03-31 16:53:42,536 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1680281622054.4920086c0c3253e0dccccde39038f1cc.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-31 16:53:42,536 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing 4920086c0c3253e0dccccde39038f1cc, disabling compactions & flushes 2023-03-31 16:53:42,536 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1680281622054.4920086c0c3253e0dccccde39038f1cc. 2023-03-31 16:53:42,536 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1680281622054.4920086c0c3253e0dccccde39038f1cc. 2023-03-31 16:53:42,536 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1680281622054.4920086c0c3253e0dccccde39038f1cc. after waiting 0 ms 2023-03-31 16:53:42,536 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1680281622054.4920086c0c3253e0dccccde39038f1cc. 2023-03-31 16:53:42,536 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1680281622054.4920086c0c3253e0dccccde39038f1cc. 2023-03-31 16:53:42,536 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for 4920086c0c3253e0dccccde39038f1cc: 2023-03-31 16:53:42,542 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-03-31 16:53:42,558 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1680281622054.4920086c0c3253e0dccccde39038f1cc.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1680281622545"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680281622545"}]},"ts":"1680281622545"} 2023-03-31 16:53:42,585 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-31 16:53:42,587 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-31 16:53:42,594 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680281622588"}]},"ts":"1680281622588"} 2023-03-31 16:53:42,598 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-03-31 16:53:42,610 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=4920086c0c3253e0dccccde39038f1cc, ASSIGN}] 2023-03-31 16:53:42,615 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=4920086c0c3253e0dccccde39038f1cc, ASSIGN 2023-03-31 16:53:42,617 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=4920086c0c3253e0dccccde39038f1cc, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,33441,1680281619375; forceNewPlan=false, retain=false 2023-03-31 16:53:42,772 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=4920086c0c3253e0dccccde39038f1cc, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:42,772 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1680281622054.4920086c0c3253e0dccccde39038f1cc.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1680281622771"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680281622771"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680281622771"}]},"ts":"1680281622771"} 2023-03-31 16:53:42,779 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 4920086c0c3253e0dccccde39038f1cc, server=jenkins-hbase4.apache.org,33441,1680281619375}] 2023-03-31 16:53:42,940 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1680281622054.4920086c0c3253e0dccccde39038f1cc. 2023-03-31 16:53:42,940 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 4920086c0c3253e0dccccde39038f1cc, NAME => 'hbase:namespace,,1680281622054.4920086c0c3253e0dccccde39038f1cc.', STARTKEY => '', ENDKEY => ''} 2023-03-31 16:53:42,941 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 4920086c0c3253e0dccccde39038f1cc 2023-03-31 16:53:42,941 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1680281622054.4920086c0c3253e0dccccde39038f1cc.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-31 16:53:42,941 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 4920086c0c3253e0dccccde39038f1cc 2023-03-31 16:53:42,941 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 4920086c0c3253e0dccccde39038f1cc 2023-03-31 16:53:42,944 INFO [StoreOpener-4920086c0c3253e0dccccde39038f1cc-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 4920086c0c3253e0dccccde39038f1cc 2023-03-31 16:53:42,947 DEBUG [StoreOpener-4920086c0c3253e0dccccde39038f1cc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/hbase/namespace/4920086c0c3253e0dccccde39038f1cc/info 2023-03-31 16:53:42,947 DEBUG [StoreOpener-4920086c0c3253e0dccccde39038f1cc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/hbase/namespace/4920086c0c3253e0dccccde39038f1cc/info 2023-03-31 16:53:42,948 INFO [StoreOpener-4920086c0c3253e0dccccde39038f1cc-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4920086c0c3253e0dccccde39038f1cc columnFamilyName info 2023-03-31 16:53:42,949 INFO [StoreOpener-4920086c0c3253e0dccccde39038f1cc-1] regionserver.HStore(310): Store=4920086c0c3253e0dccccde39038f1cc/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-31 16:53:42,950 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/hbase/namespace/4920086c0c3253e0dccccde39038f1cc 2023-03-31 16:53:42,951 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/hbase/namespace/4920086c0c3253e0dccccde39038f1cc 2023-03-31 16:53:42,956 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 4920086c0c3253e0dccccde39038f1cc 2023-03-31 16:53:42,960 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/hbase/namespace/4920086c0c3253e0dccccde39038f1cc/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-31 16:53:42,961 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 4920086c0c3253e0dccccde39038f1cc; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73325401, jitterRate=0.09263361990451813}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-31 16:53:42,961 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 4920086c0c3253e0dccccde39038f1cc: 2023-03-31 16:53:42,963 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1680281622054.4920086c0c3253e0dccccde39038f1cc., pid=6, masterSystemTime=1680281622933 2023-03-31 16:53:42,966 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1680281622054.4920086c0c3253e0dccccde39038f1cc. 2023-03-31 16:53:42,967 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1680281622054.4920086c0c3253e0dccccde39038f1cc. 2023-03-31 16:53:42,968 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=4920086c0c3253e0dccccde39038f1cc, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:42,969 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1680281622054.4920086c0c3253e0dccccde39038f1cc.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1680281622967"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680281622967"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680281622967"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680281622967"}]},"ts":"1680281622967"} 2023-03-31 16:53:42,975 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-03-31 16:53:42,975 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 4920086c0c3253e0dccccde39038f1cc, server=jenkins-hbase4.apache.org,33441,1680281619375 in 193 msec 2023-03-31 16:53:42,979 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-03-31 16:53:42,979 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=4920086c0c3253e0dccccde39038f1cc, ASSIGN in 365 msec 2023-03-31 16:53:42,980 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-31 16:53:42,981 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680281622980"}]},"ts":"1680281622980"} 2023-03-31 16:53:42,983 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-03-31 16:53:42,987 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-03-31 16:53:42,989 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 932 msec 2023-03-31 16:53:43,066 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-03-31 16:53:43,068 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-03-31 16:53:43,068 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-31 16:53:43,103 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-03-31 16:53:43,119 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-03-31 16:53:43,124 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 28 msec 2023-03-31 16:53:43,137 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-03-31 16:53:43,148 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-03-31 16:53:43,154 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 17 msec 2023-03-31 16:53:43,162 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-03-31 16:53:43,168 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-03-31 16:53:43,169 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 3.717sec 2023-03-31 16:53:43,170 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-03-31 16:53:43,172 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-03-31 16:53:43,172 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-03-31 16:53:43,173 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,43681,1680281617954-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-03-31 16:53:43,174 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,43681,1680281617954-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-03-31 16:53:43,184 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-03-31 16:53:43,222 DEBUG [Listener at localhost/44135] zookeeper.ReadOnlyZKClient(139): Connect 0x6ed65498 to 127.0.0.1:51916 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 16:53:43,226 DEBUG [Listener at localhost/44135] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@f3002be, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 16:53:43,239 DEBUG [hconnection-0x14c3948e-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 16:53:43,250 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45908, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 16:53:43,261 INFO [Listener at localhost/44135] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,43681,1680281617954 2023-03-31 16:53:43,284 INFO [Listener at localhost/44135] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=206, OpenFileDescriptor=470, MaxFileDescriptor=60000, SystemLoadAverage=333, ProcessCount=169, AvailableMemoryMB=4444 2023-03-31 16:53:43,301 DEBUG [Listener at localhost/44135] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-31 16:53:43,304 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59922, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-31 16:53:43,312 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-31 16:53:43,316 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-31 16:53:43,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-31 16:53:43,320 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-31 16:53:43,322 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-31 16:53:43,325 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4 2023-03-31 16:53:43,325 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 9 2023-03-31 16:53:43,326 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4 empty. 2023-03-31 16:53:43,327 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4 2023-03-31 16:53:43,327 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-31 16:53:43,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-31 16:53:43,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-31 16:53:43,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-31 16:53:43,744 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-31 16:53:43,746 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => e94df50bb46b68756c6a704332e1c5f4, NAME => 'TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp 2023-03-31 16:53:43,758 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-31 16:53:43,758 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing e94df50bb46b68756c6a704332e1c5f4, disabling compactions & flushes 2023-03-31 16:53:43,758 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:43,758 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:43,758 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. after waiting 0 ms 2023-03-31 16:53:43,758 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:43,758 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:43,758 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for e94df50bb46b68756c6a704332e1c5f4: 2023-03-31 16:53:43,762 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-31 16:53:43,764 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680281623763"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680281623763"}]},"ts":"1680281623763"} 2023-03-31 16:53:43,766 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-31 16:53:43,768 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-31 16:53:43,768 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680281623768"}]},"ts":"1680281623768"} 2023-03-31 16:53:43,770 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-31 16:53:43,774 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e94df50bb46b68756c6a704332e1c5f4, ASSIGN}] 2023-03-31 16:53:43,776 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e94df50bb46b68756c6a704332e1c5f4, ASSIGN 2023-03-31 16:53:43,777 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=e94df50bb46b68756c6a704332e1c5f4, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,33441,1680281619375; forceNewPlan=false, retain=false 2023-03-31 16:53:43,928 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=e94df50bb46b68756c6a704332e1c5f4, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:43,929 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680281623928"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680281623928"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680281623928"}]},"ts":"1680281623928"} 2023-03-31 16:53:43,932 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375}] 2023-03-31 16:53:43,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-31 16:53:44,091 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:44,091 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => e94df50bb46b68756c6a704332e1c5f4, NAME => 'TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.', STARTKEY => '', ENDKEY => ''} 2023-03-31 16:53:44,092 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees e94df50bb46b68756c6a704332e1c5f4 2023-03-31 16:53:44,092 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-31 16:53:44,092 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for e94df50bb46b68756c6a704332e1c5f4 2023-03-31 16:53:44,092 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for e94df50bb46b68756c6a704332e1c5f4 2023-03-31 16:53:44,094 INFO [StoreOpener-e94df50bb46b68756c6a704332e1c5f4-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region e94df50bb46b68756c6a704332e1c5f4 2023-03-31 16:53:44,097 DEBUG [StoreOpener-e94df50bb46b68756c6a704332e1c5f4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A 2023-03-31 16:53:44,097 DEBUG [StoreOpener-e94df50bb46b68756c6a704332e1c5f4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A 2023-03-31 16:53:44,098 INFO [StoreOpener-e94df50bb46b68756c6a704332e1c5f4-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-31 16:53:44,099 INFO [StoreOpener-e94df50bb46b68756c6a704332e1c5f4-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e94df50bb46b68756c6a704332e1c5f4 columnFamilyName A 2023-03-31 16:53:44,100 INFO [StoreOpener-e94df50bb46b68756c6a704332e1c5f4-1] regionserver.HStore(310): Store=e94df50bb46b68756c6a704332e1c5f4/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-31 16:53:44,101 INFO [StoreOpener-e94df50bb46b68756c6a704332e1c5f4-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region e94df50bb46b68756c6a704332e1c5f4 2023-03-31 16:53:44,103 DEBUG [StoreOpener-e94df50bb46b68756c6a704332e1c5f4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B 2023-03-31 16:53:44,103 DEBUG [StoreOpener-e94df50bb46b68756c6a704332e1c5f4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B 2023-03-31 16:53:44,103 INFO [StoreOpener-e94df50bb46b68756c6a704332e1c5f4-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-31 16:53:44,104 INFO [StoreOpener-e94df50bb46b68756c6a704332e1c5f4-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e94df50bb46b68756c6a704332e1c5f4 columnFamilyName B 2023-03-31 16:53:44,104 INFO [StoreOpener-e94df50bb46b68756c6a704332e1c5f4-1] regionserver.HStore(310): Store=e94df50bb46b68756c6a704332e1c5f4/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-31 16:53:44,105 INFO [StoreOpener-e94df50bb46b68756c6a704332e1c5f4-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region e94df50bb46b68756c6a704332e1c5f4 2023-03-31 16:53:44,108 DEBUG [StoreOpener-e94df50bb46b68756c6a704332e1c5f4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C 2023-03-31 16:53:44,108 DEBUG [StoreOpener-e94df50bb46b68756c6a704332e1c5f4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C 2023-03-31 16:53:44,108 INFO [StoreOpener-e94df50bb46b68756c6a704332e1c5f4-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-31 16:53:44,109 INFO [StoreOpener-e94df50bb46b68756c6a704332e1c5f4-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e94df50bb46b68756c6a704332e1c5f4 columnFamilyName C 2023-03-31 16:53:44,110 INFO [StoreOpener-e94df50bb46b68756c6a704332e1c5f4-1] regionserver.HStore(310): Store=e94df50bb46b68756c6a704332e1c5f4/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-31 16:53:44,111 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:44,114 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4 2023-03-31 16:53:44,114 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4 2023-03-31 16:53:44,118 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-31 16:53:44,120 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for e94df50bb46b68756c6a704332e1c5f4 2023-03-31 16:53:44,124 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-31 16:53:44,125 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened e94df50bb46b68756c6a704332e1c5f4; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64212092, jitterRate=-0.04316526651382446}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-31 16:53:44,125 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for e94df50bb46b68756c6a704332e1c5f4: 2023-03-31 16:53:44,127 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4., pid=11, masterSystemTime=1680281624084 2023-03-31 16:53:44,130 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:44,130 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:44,131 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=e94df50bb46b68756c6a704332e1c5f4, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:44,131 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680281624130"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680281624130"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680281624130"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680281624130"}]},"ts":"1680281624130"} 2023-03-31 16:53:44,137 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-03-31 16:53:44,138 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 in 202 msec 2023-03-31 16:53:44,141 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-03-31 16:53:44,141 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=e94df50bb46b68756c6a704332e1c5f4, ASSIGN in 364 msec 2023-03-31 16:53:44,143 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-31 16:53:44,143 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680281624143"}]},"ts":"1680281624143"} 2023-03-31 16:53:44,145 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-31 16:53:44,149 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-31 16:53:44,151 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 833 msec 2023-03-31 16:53:44,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-31 16:53:44,450 INFO [Listener at localhost/44135] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 9 completed 2023-03-31 16:53:44,454 DEBUG [Listener at localhost/44135] zookeeper.ReadOnlyZKClient(139): Connect 0x2678d495 to 127.0.0.1:51916 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 16:53:44,459 DEBUG [Listener at localhost/44135] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@8c3f619, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 16:53:44,462 DEBUG [Listener at localhost/44135] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 16:53:44,466 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54582, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 16:53:44,469 DEBUG [Listener at localhost/44135] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-31 16:53:44,472 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52460, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-31 16:53:44,481 DEBUG [Listener at localhost/44135] zookeeper.ReadOnlyZKClient(139): Connect 0x1b886c8f to 127.0.0.1:51916 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 16:53:44,485 DEBUG [Listener at localhost/44135] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@748f869b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 16:53:44,486 DEBUG [Listener at localhost/44135] zookeeper.ReadOnlyZKClient(139): Connect 0x0e16d41b to 127.0.0.1:51916 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 16:53:44,493 DEBUG [Listener at localhost/44135] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5e91d243, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 16:53:44,493 DEBUG [Listener at localhost/44135] zookeeper.ReadOnlyZKClient(139): Connect 0x77648431 to 127.0.0.1:51916 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 16:53:44,497 DEBUG [Listener at localhost/44135] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@30eb7fb2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 16:53:44,498 DEBUG [Listener at localhost/44135] zookeeper.ReadOnlyZKClient(139): Connect 0x7e728705 to 127.0.0.1:51916 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 16:53:44,501 DEBUG [Listener at localhost/44135] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@46192d17, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 16:53:44,502 DEBUG [Listener at localhost/44135] zookeeper.ReadOnlyZKClient(139): Connect 0x09a575b8 to 127.0.0.1:51916 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 16:53:44,505 DEBUG [Listener at localhost/44135] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2c57e1b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 16:53:44,506 DEBUG [Listener at localhost/44135] zookeeper.ReadOnlyZKClient(139): Connect 0x1818ad0a to 127.0.0.1:51916 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 16:53:44,510 DEBUG [Listener at localhost/44135] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@49d71120, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 16:53:44,510 DEBUG [Listener at localhost/44135] zookeeper.ReadOnlyZKClient(139): Connect 0x04fb2b82 to 127.0.0.1:51916 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 16:53:44,537 DEBUG [Listener at localhost/44135] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@59d6b6ea, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 16:53:44,538 DEBUG [Listener at localhost/44135] zookeeper.ReadOnlyZKClient(139): Connect 0x6129209e to 127.0.0.1:51916 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 16:53:44,541 DEBUG [Listener at localhost/44135] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@d24bbb7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 16:53:44,542 DEBUG [Listener at localhost/44135] zookeeper.ReadOnlyZKClient(139): Connect 0x4b02eeb3 to 127.0.0.1:51916 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 16:53:44,545 DEBUG [Listener at localhost/44135] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@63871a9d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 16:53:44,551 DEBUG [hconnection-0x5b5f7a77-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 16:53:44,552 DEBUG [hconnection-0x3bac190f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 16:53:44,558 DEBUG [hconnection-0x26f80c4e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 16:53:44,558 DEBUG [hconnection-0x527b0319-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 16:53:44,561 DEBUG [hconnection-0x544eb923-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 16:53:44,561 DEBUG [hconnection-0x7c8e030b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 16:53:44,561 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54602, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 16:53:44,561 DEBUG [hconnection-0x2a81f259-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 16:53:44,561 DEBUG [hconnection-0x1bd47f2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 16:53:44,563 DEBUG [hconnection-0x55d4bc5a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 16:53:44,563 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54598, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 16:53:44,573 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54608, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 16:53:44,574 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54604, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 16:53:44,574 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54618, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 16:53:44,575 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54638, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 16:53:44,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:53:44,578 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54630, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 16:53:44,580 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54628, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 16:53:44,580 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54654, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 16:53:44,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:53:44,607 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:53:44,608 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:44,608 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:53:44,608 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:53:44,611 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:44,611 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:44,613 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:44,613 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:44,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:44,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:44,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:44,617 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:44,617 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:53:44,618 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:44,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:53:44,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:44,619 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:53:44,628 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:53:44,633 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:53:44,633 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:44,637 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:53:44,642 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:53:44,642 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:53:44,642 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:44,646 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. started... 2023-03-31 16:53:44,651 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e94df50bb46b68756c6a704332e1c5f4 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-31 16:53:44,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on e94df50bb46b68756c6a704332e1c5f4 2023-03-31 16:53:44,660 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. as already flushing 2023-03-31 16:53:44,672 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=A 2023-03-31 16:53:44,673 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:53:44,674 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=B 2023-03-31 16:53:44,674 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:53:44,674 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=C 2023-03-31 16:53:44,674 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:53:44,806 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/A/010cb846e12645c2ab85d0cc9628bbd3 2023-03-31 16:53:44,826 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:44,826 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:44,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54604 deadline: 1680281684803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:44,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54630 deadline: 1680281684806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:44,829 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680281624796, max timestamp=1680281624808Num uniques 859; 2023-03-31 16:53:44,829 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 16:53:44,838 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680281624796, max timestamp=1680281624814Num uniques 859; 2023-03-31 16:53:44,839 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-03-31 16:53:44,838 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680281624796, max timestamp=1680281624814Num uniques 859; 2023-03-31 16:53:44,839 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-03-31 16:53:44,898 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:44,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54638 deadline: 1680281684895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:44,901 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:44,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54628 deadline: 1680281684898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:44,902 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:44,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54602 deadline: 1680281684898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:44,970 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:44,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54604 deadline: 1680281684968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:44,973 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:44,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54630 deadline: 1680281684971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:45,003 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:45,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54638 deadline: 1680281685002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:45,006 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:45,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54628 deadline: 1680281685006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:45,010 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:45,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54602 deadline: 1680281685006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:45,176 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:45,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54604 deadline: 1680281685175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:45,181 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:45,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54630 deadline: 1680281685177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:45,210 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:45,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54638 deadline: 1680281685209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:45,212 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:45,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54628 deadline: 1680281685211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:45,217 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:45,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54602 deadline: 1680281685215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:45,347 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/B/88d4d41c3eff4a828ff723e2c76501b5 2023-03-31 16:53:45,415 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/C/8a749e81471b4d64b888c5dc21568044 2023-03-31 16:53:45,431 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/A/010cb846e12645c2ab85d0cc9628bbd3 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/010cb846e12645c2ab85d0cc9628bbd3 2023-03-31 16:53:45,446 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/010cb846e12645c2ab85d0cc9628bbd3, entries=150, sequenceid=12, filesize=11.6 K 2023-03-31 16:53:45,451 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-03-31 16:53:45,454 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/B/88d4d41c3eff4a828ff723e2c76501b5 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/88d4d41c3eff4a828ff723e2c76501b5 2023-03-31 16:53:45,473 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/88d4d41c3eff4a828ff723e2c76501b5, entries=150, sequenceid=12, filesize=11.6 K 2023-03-31 16:53:45,486 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/C/8a749e81471b4d64b888c5dc21568044 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/8a749e81471b4d64b888c5dc21568044 2023-03-31 16:53:45,502 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/8a749e81471b4d64b888c5dc21568044, entries=150, sequenceid=12, filesize=11.6 K 2023-03-31 16:53:45,506 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=133.39 KB/136590 for e94df50bb46b68756c6a704332e1c5f4 in 855ms, sequenceid=12, compaction requested=false 2023-03-31 16:53:45,509 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-03-31 16:53:45,511 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e94df50bb46b68756c6a704332e1c5f4: 2023-03-31 16:53:45,511 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:45,512 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:53:45,513 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:53:45,513 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:45,513 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:53:45,513 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:53:45,517 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:45,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:45,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:45,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:45,524 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:45,525 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:45,525 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:53:45,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:45,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:45,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:45,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:45,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:45,528 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:53:45,533 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:53:45,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f1b95ef[Count = 0] remaining members to acquire global barrier 2023-03-31 16:53:45,533 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:45,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on e94df50bb46b68756c6a704332e1c5f4 2023-03-31 16:53:45,534 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e94df50bb46b68756c6a704332e1c5f4 3/3 column families, dataSize=140.10 KB heapSize=403.78 KB 2023-03-31 16:53:45,534 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=A 2023-03-31 16:53:45,535 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-31 16:53:45,535 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=B 2023-03-31 16:53:45,535 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-31 16:53:45,535 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=C 2023-03-31 16:53:45,535 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-31 16:53:45,558 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:45,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:45,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:45,559 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:53:45,559 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:45,559 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:53:45,559 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:53:45,559 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:53:45,565 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:45,565 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:53:45,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:45,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:45,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:45,566 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:45,566 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:53:45,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:45,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:45,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:45,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:45,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:45,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:45,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:45,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:53:45,577 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:53:45,577 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:53:45,577 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:53:45,577 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:53:45,577 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:53:45,577 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:53:45,585 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:45,585 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:45,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:45,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:45,585 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:45,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:45,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:45,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:45,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:45,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:45,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:45,593 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:45,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:45,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:45,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:45,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:45,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:45,596 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:45,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:45,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:45,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:45,618 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:45,618 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:45,618 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:45,618 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:53:45,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:45,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:45,618 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:45,618 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:53:45,618 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:45,621 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:45,621 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:45,621 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:45,621 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:45,621 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:45,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:45,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:45,622 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:53:45,622 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:53:45,623 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/A/a19e77acb6394df3a197abed58be738e 2023-03-31 16:53:45,623 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:45,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54638 deadline: 1680281685597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:45,624 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:45,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54602 deadline: 1680281685603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:45,655 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:45,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54604 deadline: 1680281685619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:45,656 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:45,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54630 deadline: 1680281685624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:45,694 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:45,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54628 deadline: 1680281685638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:45,722 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:53:45,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:53:45,735 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:45,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54638 deadline: 1680281685728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:45,735 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:45,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54602 deadline: 1680281685732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:45,739 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:53:45,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:53:45,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:53:45,747 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:53:45,747 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:45,753 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:53:45,754 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:53:45,754 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:45,755 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:45,756 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:45,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:45,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:45,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:45,758 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:45,760 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:45,760 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:53:45,761 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:45,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:53:45,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:45,762 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:53:45,762 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:53:45,762 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:53:45,762 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:53:45,762 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:45,763 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:53:45,763 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:53:45,763 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:53:45,763 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:45,764 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. started... 2023-03-31 16:53:45,764 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. as already flushing 2023-03-31 16:53:45,764 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:45,765 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:53:45,765 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:53:45,765 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:45,765 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:53:45,765 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:53:45,767 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:45,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:45,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:45,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:45,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:45,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:45,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:45,769 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:45,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:45,770 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:45,770 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:53:45,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:45,771 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:53:45,771 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4012c258[Count = 0] remaining members to acquire global barrier 2023-03-31 16:53:45,771 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:53:45,771 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:45,772 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:45,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54630 deadline: 1680281685760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:45,773 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:45,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54604 deadline: 1680281685761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:45,774 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:45,774 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:53:45,774 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:45,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:45,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:45,775 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:53:45,775 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:53:45,775 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:53:45,778 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:45,778 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:53:45,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:45,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:45,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:45,778 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:45,778 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:53:45,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:45,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:45,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:45,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:45,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:45,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:45,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:45,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:53:45,783 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:53:45,783 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:53:45,783 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:53:45,783 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:53:45,783 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:53:45,783 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:53:45,788 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:45,788 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:45,789 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:45,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:45,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:45,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:45,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:45,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:45,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:45,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:45,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:45,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:45,791 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:45,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:45,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:45,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:45,794 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:45,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:45,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:45,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:45,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:45,805 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:45,805 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:45,805 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:45,805 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:45,805 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:45,805 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:53:45,806 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:53:45,806 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:53:45,805 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:45,806 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:45,806 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:45,806 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:45,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:45,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:45,806 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:45,806 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:53:45,806 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:53:45,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:45,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:45,825 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:45,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54628 deadline: 1680281685805, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:45,906 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:53:45,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:53:45,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:53:45,943 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:45,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54638 deadline: 1680281685940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:45,944 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:45,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54602 deadline: 1680281685943, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:45,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:53:45,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:53:45,950 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:53:45,950 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:45,953 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:53:45,953 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:53:45,953 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:45,953 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:45,956 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:45,956 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:45,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:45,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:45,957 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:45,957 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:53:45,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:45,958 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:45,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:53:45,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:45,959 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:53:45,959 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:53:45,959 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:53:45,959 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:53:45,959 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:45,962 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:53:45,962 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:53:45,962 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:53:45,962 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:45,962 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. started... 2023-03-31 16:53:45,962 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. as already flushing 2023-03-31 16:53:45,962 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:45,963 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:53:45,963 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:53:45,963 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:45,963 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:53:45,963 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:53:45,969 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:45,969 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:45,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:45,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:45,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:45,969 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:45,969 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:53:45,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:45,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:45,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:45,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:45,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:45,972 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:53:45,972 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3164dc07[Count = 0] remaining members to acquire global barrier 2023-03-31 16:53:45,972 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:53:45,972 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:45,974 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:45,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:45,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:45,974 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:53:45,974 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:53:45,974 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:53:45,975 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:45,975 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:53:45,981 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:45,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54630 deadline: 1680281685977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:45,981 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:45,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:45,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:45,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:45,982 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:53:45,982 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:45,982 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:53:45,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:45,981 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:45,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:45,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54604 deadline: 1680281685978, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:45,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:45,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:45,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:45,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:45,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:45,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:53:45,990 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:53:45,990 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:53:45,990 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:53:45,990 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:53:45,990 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:53:45,990 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:53:45,993 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:45,993 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:45,993 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:45,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:45,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:45,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:45,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:45,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:45,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:45,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:45,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:45,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:45,995 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:45,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:45,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:45,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:46,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:46,001 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:46,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:46,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:46,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:46,009 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:46,009 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:46,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:46,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:46,009 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:46,009 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:46,009 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:46,010 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:46,010 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:53:46,010 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:53:46,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:46,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:46,010 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:46,011 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:46,011 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:46,011 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:46,011 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:53:46,011 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:53:46,034 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:46,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54628 deadline: 1680281686029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:46,100 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.57 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/B/ab68f0e33a4f4cd1b8eaccb5ec481708 2023-03-31 16:53:46,112 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:53:46,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:53:46,133 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:53:46,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:53:46,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:53:46,150 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:53:46,150 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:46,155 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:53:46,155 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:53:46,157 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:46,157 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:46,158 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:46,158 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:46,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:46,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:46,159 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:46,159 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:53:46,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:46,160 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:46,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:53:46,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:46,160 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:53:46,164 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:53:46,164 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:53:46,164 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:53:46,164 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:46,165 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:53:46,166 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:53:46,166 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:53:46,168 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:46,169 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. started... 2023-03-31 16:53:46,169 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. as already flushing 2023-03-31 16:53:46,170 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:46,172 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:53:46,172 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:53:46,172 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:46,172 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:53:46,172 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:53:46,176 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:46,176 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:46,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:46,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:46,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:46,177 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:46,177 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:53:46,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:46,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:46,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:46,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:46,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:46,180 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:53:46,180 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@588c9675[Count = 0] remaining members to acquire global barrier 2023-03-31 16:53:46,180 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:53:46,180 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:46,181 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:46,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:46,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:46,182 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:53:46,182 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:53:46,182 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:53:46,182 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:46,182 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:53:46,184 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:46,184 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:53:46,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:46,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:46,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:46,185 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:46,185 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:53:46,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:46,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:46,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:46,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:46,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:46,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:46,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:46,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:53:46,190 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:53:46,190 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:53:46,190 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:53:46,190 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:53:46,190 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:53:46,190 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:53:46,193 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:46,193 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:46,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:46,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:46,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:46,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:46,194 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:46,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:46,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:46,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:46,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:46,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:46,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:46,195 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:46,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:46,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:46,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:46,197 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:46,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:46,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:46,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:46,209 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:46,209 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:46,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:46,209 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:46,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:46,209 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:53:46,209 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:53:46,209 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:46,209 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:46,209 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:46,209 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:46,209 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:46,209 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:53:46,210 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:53:46,209 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:46,210 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:46,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:46,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:46,251 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:46,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54638 deadline: 1680281686246, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:46,268 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:46,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54602 deadline: 1680281686267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:46,286 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:46,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54630 deadline: 1680281686285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:46,288 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:46,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54604 deadline: 1680281686287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:46,310 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:53:46,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:53:46,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:53:46,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:53:46,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:53:46,320 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:53:46,320 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:46,324 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:53:46,324 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:53:46,325 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:46,325 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:46,327 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:46,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:46,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:46,327 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:46,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:46,328 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:46,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:53:46,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:46,329 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:53:46,330 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:46,330 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:53:46,330 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:53:46,330 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:53:46,330 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:53:46,330 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:46,331 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:53:46,332 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:53:46,332 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:53:46,332 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:46,332 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. started... 2023-03-31 16:53:46,332 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. as already flushing 2023-03-31 16:53:46,332 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:46,339 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:53:46,339 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:53:46,339 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:46,339 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:53:46,339 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:53:46,341 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:46,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:46,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:46,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:46,342 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:46,342 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:46,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54628 deadline: 1680281686339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:46,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:46,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:46,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:46,344 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:46,344 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:53:46,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:46,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:46,345 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:53:46,345 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b6bc466[Count = 0] remaining members to acquire global barrier 2023-03-31 16:53:46,345 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:53:46,345 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:46,348 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:46,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:46,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:46,348 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:53:46,348 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:53:46,348 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:53:46,349 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:46,349 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:53:46,351 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:46,351 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:53:46,351 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:46,351 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:53:46,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:46,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:46,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:46,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:46,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:46,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:46,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:46,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:46,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:46,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:46,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:53:46,361 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:53:46,362 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:53:46,362 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:53:46,362 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:53:46,362 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:53:46,362 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:53:46,364 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:46,364 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:46,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:46,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:46,364 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:46,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:46,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:46,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:46,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:46,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:46,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:46,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:46,366 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:46,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:46,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:46,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:46,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:46,371 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:46,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:46,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:46,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:46,379 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:46,379 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:46,379 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:46,379 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:53:46,379 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:53:46,379 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:46,379 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:46,379 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:46,380 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:46,380 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:46,379 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:46,379 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:46,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:46,380 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:53:46,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:46,384 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:53:46,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:46,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:46,485 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:53:46,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:53:46,490 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:53:46,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:53:46,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:53:46,492 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:53:46,492 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:46,493 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:53:46,493 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:53:46,493 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:46,494 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:46,495 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:46,495 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:46,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:46,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:46,496 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:46,496 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:53:46,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:46,497 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:46,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:53:46,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:46,497 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:53:46,497 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:53:46,497 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:53:46,497 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:53:46,498 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:46,498 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:53:46,499 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:53:46,499 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:53:46,499 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:46,499 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. started... 2023-03-31 16:53:46,499 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. as already flushing 2023-03-31 16:53:46,499 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:46,500 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:53:46,500 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:53:46,500 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:46,500 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:53:46,500 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:53:46,502 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:46,502 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:46,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:46,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:46,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:46,502 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:46,503 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:53:46,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:46,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:46,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:46,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:46,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:46,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:53:46,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5dbdc751[Count = 0] remaining members to acquire global barrier 2023-03-31 16:53:46,505 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:53:46,505 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:46,508 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:46,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:46,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:46,508 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:53:46,508 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:53:46,508 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:53:46,508 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:46,508 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:53:46,510 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:46,510 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:53:46,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:46,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:46,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:46,510 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:46,510 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:53:46,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:46,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:46,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:46,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:46,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:46,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:46,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:46,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:53:46,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:53:46,516 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:53:46,516 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:53:46,516 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:53:46,516 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:53:46,516 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:53:46,518 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:46,518 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:46,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:46,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:46,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:46,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:46,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:46,518 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:46,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:46,519 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:46,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:46,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:46,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:46,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:46,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:46,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:46,526 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:46,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:46,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:46,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:46,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:46,531 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:46,531 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:46,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:46,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:46,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:46,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:46,531 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:46,532 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:46,532 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:46,532 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:46,532 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:46,532 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:46,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:53:46,533 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:53:46,532 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:46,533 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:46,533 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:53:46,534 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:53:46,534 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:53:46,572 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.57 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/C/ecd77e38378244408ead7960caf26773 2023-03-31 16:53:46,585 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/A/a19e77acb6394df3a197abed58be738e as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/a19e77acb6394df3a197abed58be738e 2023-03-31 16:53:46,597 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/a19e77acb6394df3a197abed58be738e, entries=200, sequenceid=50, filesize=13.9 K 2023-03-31 16:53:46,600 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/B/ab68f0e33a4f4cd1b8eaccb5ec481708 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/ab68f0e33a4f4cd1b8eaccb5ec481708 2023-03-31 16:53:46,627 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/ab68f0e33a4f4cd1b8eaccb5ec481708, entries=150, sequenceid=50, filesize=11.6 K 2023-03-31 16:53:46,629 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/C/ecd77e38378244408ead7960caf26773 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/ecd77e38378244408ead7960caf26773 2023-03-31 16:53:46,634 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:53:46,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:53:46,645 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/ecd77e38378244408ead7960caf26773, entries=150, sequenceid=50, filesize=11.6 K 2023-03-31 16:53:46,646 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-03-31 16:53:46,647 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:53:46,653 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.10 KB/143460, heapSize ~403.73 KB/413424, currentSize=46.96 KB/48090 for e94df50bb46b68756c6a704332e1c5f4 in 1119ms, sequenceid=50, compaction requested=false 2023-03-31 16:53:46,653 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-03-31 16:53:46,656 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e94df50bb46b68756c6a704332e1c5f4: 2023-03-31 16:53:46,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:53:46,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:53:46,658 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:53:46,658 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:46,659 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:53:46,659 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:53:46,660 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:46,660 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:46,663 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:46,663 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:46,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:46,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:46,663 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:46,663 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:53:46,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:46,664 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:46,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:53:46,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:46,666 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:53:46,666 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:53:46,666 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:53:46,666 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:53:46,666 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:46,667 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:53:46,669 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:53:46,669 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:53:46,670 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:46,670 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. started... 2023-03-31 16:53:46,670 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e94df50bb46b68756c6a704332e1c5f4 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-31 16:53:46,671 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=A 2023-03-31 16:53:46,671 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:53:46,671 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=B 2023-03-31 16:53:46,671 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:53:46,671 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=C 2023-03-31 16:53:46,671 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:53:46,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on e94df50bb46b68756c6a704332e1c5f4 2023-03-31 16:53:46,760 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. as already flushing 2023-03-31 16:53:46,821 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680281626794, max timestamp=1680281626816Num uniques 859; 2023-03-31 16:53:46,821 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-31 16:53:46,833 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:46,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54630 deadline: 1680281686828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:46,834 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:46,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54604 deadline: 1680281686830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:46,839 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:46,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54602 deadline: 1680281686832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:46,839 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:46,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54638 deadline: 1680281686834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:46,852 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:46,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54628 deadline: 1680281686847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:46,941 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:46,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54630 deadline: 1680281686936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:46,942 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:46,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54604 deadline: 1680281686936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:46,943 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:46,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54602 deadline: 1680281686942, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:46,945 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:46,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54638 deadline: 1680281686942, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:47,125 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=60 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/A/46b76279f65a48fc99c1179eab1a7971 2023-03-31 16:53:47,146 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:47,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54630 deadline: 1680281687145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:47,157 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:47,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54602 deadline: 1680281687146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:47,157 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:47,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54604 deadline: 1680281687147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:47,158 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:47,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54638 deadline: 1680281687148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:47,419 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-31 16:53:47,451 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:47,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54630 deadline: 1680281687449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:47,470 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:47,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54602 deadline: 1680281687461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:47,471 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:47,471 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:47,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54638 deadline: 1680281687462, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:47,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54604 deadline: 1680281687462, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:47,577 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=60 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/B/cb55249a2b5148cfb6c0f4445c9c09cb 2023-03-31 16:53:47,863 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:47,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54628 deadline: 1680281687862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:47,958 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:47,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54630 deadline: 1680281687955, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:47,981 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:47,983 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:47,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54602 deadline: 1680281687974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:47,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54638 deadline: 1680281687978, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:47,985 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:47,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54604 deadline: 1680281687983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:48,014 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=60 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/C/8cd7d8ff21044e8e9a36ed332196f6b2 2023-03-31 16:53:48,027 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/A/46b76279f65a48fc99c1179eab1a7971 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/46b76279f65a48fc99c1179eab1a7971 2023-03-31 16:53:48,051 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/46b76279f65a48fc99c1179eab1a7971, entries=150, sequenceid=60, filesize=11.6 K 2023-03-31 16:53:48,054 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/B/cb55249a2b5148cfb6c0f4445c9c09cb as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/cb55249a2b5148cfb6c0f4445c9c09cb 2023-03-31 16:53:48,065 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/cb55249a2b5148cfb6c0f4445c9c09cb, entries=150, sequenceid=60, filesize=11.6 K 2023-03-31 16:53:48,071 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/C/8cd7d8ff21044e8e9a36ed332196f6b2 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/8cd7d8ff21044e8e9a36ed332196f6b2 2023-03-31 16:53:48,080 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/8cd7d8ff21044e8e9a36ed332196f6b2, entries=150, sequenceid=60, filesize=11.6 K 2023-03-31 16:53:48,088 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=129.71 KB/132820 for e94df50bb46b68756c6a704332e1c5f4 in 1418ms, sequenceid=60, compaction requested=true 2023-03-31 16:53:48,088 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e94df50bb46b68756c6a704332e1c5f4: 2023-03-31 16:53:48,088 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:48,089 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:53:48,089 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:53:48,089 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:48,089 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:53:48,089 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:53:48,096 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:48,096 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:48,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:48,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:48,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:48,097 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:48,097 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:53:48,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:48,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:48,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:48,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:48,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:48,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:53:48,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4aaebc05[Count = 0] remaining members to acquire global barrier 2023-03-31 16:53:48,099 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:53:48,099 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:48,101 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:48,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:48,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:48,101 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:53:48,101 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:53:48,101 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:53:48,101 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:48,102 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:53:48,103 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:53:48,103 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:48,103 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:53:48,107 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:48,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:48,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:48,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:48,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:48,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:48,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:48,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:48,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:48,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:48,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:48,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:53:48,112 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:53:48,112 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:53:48,112 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:53:48,112 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:53:48,112 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:53:48,112 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:53:48,114 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:48,115 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:48,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:48,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:48,115 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:48,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:48,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:48,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:48,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:48,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:48,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:48,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:48,117 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:48,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:48,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:48,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:48,119 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:48,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:48,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:48,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:48,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:48,126 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:48,126 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:48,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:48,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:48,126 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:48,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:53:48,128 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:53:48,126 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:48,126 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:53:48,129 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:53:48,129 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:53:48,129 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:48,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:48,129 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:48,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:48,129 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:48,129 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:48,129 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:48,129 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:48,132 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-03-31 16:53:48,134 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-03-31 16:53:48,229 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:53:48,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:53:48,232 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:53:48,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:53:48,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:53:48,237 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:53:48,239 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:48,240 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:53:48,240 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:53:48,241 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:48,241 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:48,242 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:48,242 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:48,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:48,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:48,243 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:48,243 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:53:48,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:48,244 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:48,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:53:48,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:48,245 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:53:48,245 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:53:48,245 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:53:48,245 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:53:48,245 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:48,246 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:53:48,248 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:53:48,248 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:48,248 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:53:48,248 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. started... 2023-03-31 16:53:48,248 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e94df50bb46b68756c6a704332e1c5f4 3/3 column families, dataSize=129.71 KB heapSize=376.95 KB 2023-03-31 16:53:48,248 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=A 2023-03-31 16:53:48,249 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:53:48,249 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=B 2023-03-31 16:53:48,249 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:53:48,249 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=C 2023-03-31 16:53:48,249 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:53:48,293 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=86 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/A/ec500428d0c4460dab6c901550e705e4 2023-03-31 16:53:48,328 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=86 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/B/7d5121685cce4e979af87c23f0f6475d 2023-03-31 16:53:48,758 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=86 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/C/adfdf62a70dc4f1b81c8933dc02f819e 2023-03-31 16:53:48,769 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/A/ec500428d0c4460dab6c901550e705e4 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/ec500428d0c4460dab6c901550e705e4 2023-03-31 16:53:48,787 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/ec500428d0c4460dab6c901550e705e4, entries=150, sequenceid=86, filesize=11.6 K 2023-03-31 16:53:48,790 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/B/7d5121685cce4e979af87c23f0f6475d as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/7d5121685cce4e979af87c23f0f6475d 2023-03-31 16:53:48,798 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/7d5121685cce4e979af87c23f0f6475d, entries=150, sequenceid=86, filesize=11.6 K 2023-03-31 16:53:48,800 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/C/adfdf62a70dc4f1b81c8933dc02f819e as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/adfdf62a70dc4f1b81c8933dc02f819e 2023-03-31 16:53:48,817 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/adfdf62a70dc4f1b81c8933dc02f819e, entries=150, sequenceid=86, filesize=11.6 K 2023-03-31 16:53:48,821 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~129.71 KB/132820, heapSize ~376.91 KB/385952, currentSize=0 B/0 for e94df50bb46b68756c6a704332e1c5f4 in 573ms, sequenceid=86, compaction requested=true 2023-03-31 16:53:48,821 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e94df50bb46b68756c6a704332e1c5f4: 2023-03-31 16:53:48,821 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:48,821 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:53:48,822 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:53:48,822 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:48,822 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:53:48,822 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:53:48,824 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:48,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:48,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:48,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:48,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:48,824 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:48,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:48,828 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:48,828 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:53:48,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:48,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:48,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:48,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:53:48,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76adb41b[Count = 0] remaining members to acquire global barrier 2023-03-31 16:53:48,836 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:53:48,836 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:48,838 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:48,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:48,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:48,839 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:53:48,839 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:53:48,839 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:53:48,840 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:48,840 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:53:48,842 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:53:48,842 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:48,842 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:53:48,845 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:48,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:48,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:48,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:48,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:48,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:48,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:48,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:48,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:48,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:48,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:48,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:53:48,851 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:53:48,851 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:53:48,851 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:53:48,851 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:53:48,851 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:53:48,851 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:53:48,853 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:48,853 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:48,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:48,853 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:48,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:48,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:48,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:48,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:48,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:48,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:48,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:48,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:48,855 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:48,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:48,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:48,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:48,858 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:48,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:48,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:48,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:48,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:48,869 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:48,869 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:48,869 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:48,869 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:48,870 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:48,870 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:48,870 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:48,870 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:48,871 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:53:48,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:53:48,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:53:48,871 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:48,872 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:48,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:48,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:48,872 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:53:48,872 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:53:48,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:48,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:48,972 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:53:48,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:53:48,978 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:53:49,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on e94df50bb46b68756c6a704332e1c5f4 2023-03-31 16:53:49,006 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e94df50bb46b68756c6a704332e1c5f4 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-31 16:53:49,006 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=A 2023-03-31 16:53:49,007 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:53:49,007 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=B 2023-03-31 16:53:49,007 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:53:49,007 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=C 2023-03-31 16:53:49,007 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:53:49,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:53:49,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:53:49,012 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:53:49,016 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:49,016 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:53:49,017 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:53:49,018 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:49,019 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:49,021 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:49,021 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:49,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:49,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:49,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:49,022 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:49,022 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:53:49,022 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:49,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:53:49,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:49,023 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:53:49,037 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:53:49,041 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:53:49,041 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:53:49,041 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:49,042 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:53:49,047 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:53:49,048 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:53:49,048 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:49,048 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. started... 2023-03-31 16:53:49,049 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. as already flushing 2023-03-31 16:53:49,049 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:49,049 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:53:49,049 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:53:49,049 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:49,049 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:53:49,049 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:53:49,055 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=96 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/A/15cfa5e5c30442e2a7602aa3f38a1295 2023-03-31 16:53:49,056 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:49,056 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:49,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:49,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:49,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:49,057 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:49,057 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:53:49,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:49,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:49,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:49,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:49,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:49,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:53:49,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23b0065c[Count = 0] remaining members to acquire global barrier 2023-03-31 16:53:49,061 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:53:49,061 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:49,063 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:49,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:49,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:49,064 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:53:49,064 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:53:49,064 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:53:49,066 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:53:49,066 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:49,066 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:49,066 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:53:49,068 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:53:49,069 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:53:49,069 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:53:49,069 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:53:49,069 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:53:49,069 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:53:49,069 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:53:49,071 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:49,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:49,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:49,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:49,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:49,072 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:49,072 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:49,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:49,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:49,072 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:49,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:49,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:49,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:49,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:49,075 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:49,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:49,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:49,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:49,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:49,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:49,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:49,082 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:49,082 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:49,082 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:49,082 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:49,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:49,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:49,083 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:49,083 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:49,083 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:49,083 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:53:49,083 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:53:49,083 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:49,083 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:49,083 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:49,083 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:53:49,083 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:53:49,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:49,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:49,088 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680281629053, max timestamp=1680281629077Num uniques 859; 2023-03-31 16:53:49,089 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 16:53:49,089 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680281629053, max timestamp=1680281629077Num uniques 859; 2023-03-31 16:53:49,089 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 16:53:49,089 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680281629053, max timestamp=1680281629077Num uniques 859; 2023-03-31 16:53:49,089 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 16:53:49,135 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=96 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/B/e3bd8fd57c824772af50067e773882c9 2023-03-31 16:53:49,144 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:49,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54630 deadline: 1680281689134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:49,145 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:49,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54638 deadline: 1680281689137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:49,147 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:49,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54604 deadline: 1680281689142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:49,153 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:49,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54602 deadline: 1680281689145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:49,183 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:53:49,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:53:49,188 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:53:49,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:53:49,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:53:49,240 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:53:49,241 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:49,240 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=96 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/C/fa8e33de7b824c9a8966ff6a0151e6e4 2023-03-31 16:53:49,242 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:53:49,243 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:53:49,243 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:49,244 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:49,245 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:49,245 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:49,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:49,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:49,246 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:49,246 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:53:49,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:49,248 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:49,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:53:49,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:49,249 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:53:49,249 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:49,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54638 deadline: 1680281689248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:49,250 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:49,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54604 deadline: 1680281689249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:49,250 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:53:49,250 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:53:49,251 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:53:49,251 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:49,254 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:53:49,254 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:53:49,254 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:53:49,255 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:49,255 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. started... 2023-03-31 16:53:49,255 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. as already flushing 2023-03-31 16:53:49,255 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:49,255 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:53:49,255 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:53:49,255 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:49,255 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:53:49,255 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:53:49,256 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:49,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54602 deadline: 1680281689255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:49,257 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:49,257 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:49,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:49,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:49,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:49,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:49,258 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:49,258 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:53:49,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:49,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:49,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:49,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:49,261 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:53:49,261 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1004ca34[Count = 0] remaining members to acquire global barrier 2023-03-31 16:53:49,261 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:53:49,261 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:49,264 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:49,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:49,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:49,264 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:53:49,264 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:53:49,264 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:53:49,264 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/A/15cfa5e5c30442e2a7602aa3f38a1295 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/15cfa5e5c30442e2a7602aa3f38a1295 2023-03-31 16:53:49,264 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:49,264 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:53:49,266 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:53:49,266 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:49,266 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:53:49,266 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:49,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:49,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:49,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:49,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:49,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:49,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:49,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:49,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:49,276 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:49,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54630 deadline: 1680281689272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:49,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:49,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:49,280 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/15cfa5e5c30442e2a7602aa3f38a1295, entries=150, sequenceid=96, filesize=11.6 K 2023-03-31 16:53:49,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:53:49,282 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:53:49,282 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:53:49,282 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:53:49,282 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:53:49,282 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:53:49,283 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:53:49,284 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/B/e3bd8fd57c824772af50067e773882c9 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/e3bd8fd57c824772af50067e773882c9 2023-03-31 16:53:49,285 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:49,285 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:49,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:49,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:49,286 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:49,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:49,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:49,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:49,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:49,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:49,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:49,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:49,289 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:49,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:49,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:49,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:49,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:49,291 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:49,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:49,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:49,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:49,303 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:49,303 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:49,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:49,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:49,303 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:49,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:53:49,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:53:49,303 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:49,303 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:49,304 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:49,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:49,304 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:49,304 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:53:49,304 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:49,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:49,304 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:49,304 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:49,304 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:53:49,306 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/e3bd8fd57c824772af50067e773882c9, entries=150, sequenceid=96, filesize=11.6 K 2023-03-31 16:53:49,308 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/C/fa8e33de7b824c9a8966ff6a0151e6e4 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/fa8e33de7b824c9a8966ff6a0151e6e4 2023-03-31 16:53:49,353 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/fa8e33de7b824c9a8966ff6a0151e6e4, entries=150, sequenceid=96, filesize=11.6 K 2023-03-31 16:53:49,355 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for e94df50bb46b68756c6a704332e1c5f4 in 349ms, sequenceid=96, compaction requested=true 2023-03-31 16:53:49,355 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e94df50bb46b68756c6a704332e1c5f4: 2023-03-31 16:53:49,358 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:53:49,358 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:53:49,358 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:53:49,359 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-31 16:53:49,359 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-31 16:53:49,366 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61770 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-31 16:53:49,366 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59430 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-31 16:53:49,369 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1912): e94df50bb46b68756c6a704332e1c5f4/A is initiating minor compaction (all files) 2023-03-31 16:53:49,369 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): e94df50bb46b68756c6a704332e1c5f4/B is initiating minor compaction (all files) 2023-03-31 16:53:49,369 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e94df50bb46b68756c6a704332e1c5f4/A in TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:49,369 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e94df50bb46b68756c6a704332e1c5f4/B in TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:49,369 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/88d4d41c3eff4a828ff723e2c76501b5, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/ab68f0e33a4f4cd1b8eaccb5ec481708, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/cb55249a2b5148cfb6c0f4445c9c09cb, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/7d5121685cce4e979af87c23f0f6475d, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/e3bd8fd57c824772af50067e773882c9] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp, totalSize=58.0 K 2023-03-31 16:53:49,369 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/010cb846e12645c2ab85d0cc9628bbd3, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/a19e77acb6394df3a197abed58be738e, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/46b76279f65a48fc99c1179eab1a7971, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/ec500428d0c4460dab6c901550e705e4, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/15cfa5e5c30442e2a7602aa3f38a1295] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp, totalSize=60.3 K 2023-03-31 16:53:49,371 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 88d4d41c3eff4a828ff723e2c76501b5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1680281624622 2023-03-31 16:53:49,373 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 010cb846e12645c2ab85d0cc9628bbd3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1680281624622 2023-03-31 16:53:49,377 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting a19e77acb6394df3a197abed58be738e, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1680281624859 2023-03-31 16:53:49,378 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting ab68f0e33a4f4cd1b8eaccb5ec481708, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1680281624859 2023-03-31 16:53:49,380 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting cb55249a2b5148cfb6c0f4445c9c09cb, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1680281625578 2023-03-31 16:53:49,380 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 7d5121685cce4e979af87c23f0f6475d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=86, earliestPutTs=1680281626827 2023-03-31 16:53:49,381 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting e3bd8fd57c824772af50067e773882c9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1680281628988 2023-03-31 16:53:49,382 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 46b76279f65a48fc99c1179eab1a7971, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1680281625578 2023-03-31 16:53:49,401 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting ec500428d0c4460dab6c901550e705e4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=86, earliestPutTs=1680281626827 2023-03-31 16:53:49,403 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 15cfa5e5c30442e2a7602aa3f38a1295, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1680281628988 2023-03-31 16:53:49,404 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:53:49,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:53:49,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:53:49,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:53:49,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:53:49,423 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:53:49,423 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:49,425 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:53:49,425 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:53:49,425 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:49,425 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:49,427 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:49,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:49,427 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:49,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:49,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:49,428 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:49,428 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:53:49,428 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:49,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:53:49,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:49,429 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:53:49,432 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:53:49,432 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:53:49,432 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:53:49,432 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:49,436 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): e94df50bb46b68756c6a704332e1c5f4#B#compaction#15 average throughput is 0.73 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:53:49,441 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:53:49,441 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:53:49,441 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:53:49,444 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:49,444 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. started... 2023-03-31 16:53:49,444 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e94df50bb46b68756c6a704332e1c5f4 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-03-31 16:53:49,444 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=A 2023-03-31 16:53:49,445 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:53:49,445 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=B 2023-03-31 16:53:49,445 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:53:49,445 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=C 2023-03-31 16:53:49,445 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:53:49,446 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] throttle.PressureAwareThroughputController(145): e94df50bb46b68756c6a704332e1c5f4#A#compaction#16 average throughput is 0.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:53:49,465 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. as already flushing 2023-03-31 16:53:49,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on e94df50bb46b68756c6a704332e1c5f4 2023-03-31 16:53:49,522 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/B/1f1c599c73c840b584187ea68eb50fc4 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/1f1c599c73c840b584187ea68eb50fc4 2023-03-31 16:53:49,533 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=133 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/A/c447ed0d8629424e93db831f3698ec0e 2023-03-31 16:53:49,536 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:49,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54604 deadline: 1680281689507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:49,544 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:49,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54630 deadline: 1680281689510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:49,545 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:49,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54602 deadline: 1680281689516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:49,546 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:49,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54638 deadline: 1680281689532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:49,557 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in e94df50bb46b68756c6a704332e1c5f4/B of e94df50bb46b68756c6a704332e1c5f4 into 1f1c599c73c840b584187ea68eb50fc4(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:53:49,557 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e94df50bb46b68756c6a704332e1c5f4: 2023-03-31 16:53:49,557 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4., storeName=e94df50bb46b68756c6a704332e1c5f4/B, priority=11, startTime=1680281629358; duration=0sec 2023-03-31 16:53:49,563 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:53:49,563 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-31 16:53:49,570 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59430 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-31 16:53:49,570 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): e94df50bb46b68756c6a704332e1c5f4/C is initiating minor compaction (all files) 2023-03-31 16:53:49,570 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e94df50bb46b68756c6a704332e1c5f4/C in TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:49,571 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/8a749e81471b4d64b888c5dc21568044, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/ecd77e38378244408ead7960caf26773, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/8cd7d8ff21044e8e9a36ed332196f6b2, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/adfdf62a70dc4f1b81c8933dc02f819e, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/fa8e33de7b824c9a8966ff6a0151e6e4] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp, totalSize=58.0 K 2023-03-31 16:53:49,571 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 8a749e81471b4d64b888c5dc21568044, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1680281624622 2023-03-31 16:53:49,575 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting ecd77e38378244408ead7960caf26773, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1680281624859 2023-03-31 16:53:49,594 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 8cd7d8ff21044e8e9a36ed332196f6b2, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1680281625578 2023-03-31 16:53:49,596 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting adfdf62a70dc4f1b81c8933dc02f819e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=86, earliestPutTs=1680281626827 2023-03-31 16:53:49,597 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting fa8e33de7b824c9a8966ff6a0151e6e4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1680281628988 2023-03-31 16:53:49,598 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=133 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/B/db2a8599b6cb4821bce4873cfd83b9f5 2023-03-31 16:53:49,637 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): e94df50bb46b68756c6a704332e1c5f4#C#compaction#20 average throughput is 0.66 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:53:49,656 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=133 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/C/8da43654ffe04f1f9168f9c43e5e3263 2023-03-31 16:53:49,657 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:49,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54630 deadline: 1680281689647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:49,657 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:49,657 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:49,657 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:49,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54638 deadline: 1680281689650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:49,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54604 deadline: 1680281689646, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:49,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54602 deadline: 1680281689649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:49,670 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/A/c447ed0d8629424e93db831f3698ec0e as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/c447ed0d8629424e93db831f3698ec0e 2023-03-31 16:53:49,684 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/c447ed0d8629424e93db831f3698ec0e, entries=150, sequenceid=133, filesize=11.8 K 2023-03-31 16:53:49,686 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/B/db2a8599b6cb4821bce4873cfd83b9f5 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/db2a8599b6cb4821bce4873cfd83b9f5 2023-03-31 16:53:49,700 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/C/1ce9b06377fe4597b18ab6b3bcf0ee7c as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/1ce9b06377fe4597b18ab6b3bcf0ee7c 2023-03-31 16:53:49,708 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/db2a8599b6cb4821bce4873cfd83b9f5, entries=150, sequenceid=133, filesize=11.8 K 2023-03-31 16:53:49,716 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/C/8da43654ffe04f1f9168f9c43e5e3263 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/8da43654ffe04f1f9168f9c43e5e3263 2023-03-31 16:53:49,724 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in e94df50bb46b68756c6a704332e1c5f4/C of e94df50bb46b68756c6a704332e1c5f4 into 1ce9b06377fe4597b18ab6b3bcf0ee7c(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:53:49,725 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e94df50bb46b68756c6a704332e1c5f4: 2023-03-31 16:53:49,725 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4., storeName=e94df50bb46b68756c6a704332e1c5f4/C, priority=11, startTime=1680281629358; duration=0sec 2023-03-31 16:53:49,725 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:53:49,739 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/8da43654ffe04f1f9168f9c43e5e3263, entries=150, sequenceid=133, filesize=11.8 K 2023-03-31 16:53:49,743 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for e94df50bb46b68756c6a704332e1c5f4 in 298ms, sequenceid=133, compaction requested=false 2023-03-31 16:53:49,743 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e94df50bb46b68756c6a704332e1c5f4: 2023-03-31 16:53:49,743 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:49,743 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:53:49,743 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:53:49,743 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:49,743 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:53:49,743 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:53:49,745 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:49,745 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:49,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:49,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:49,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:49,745 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:49,745 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:53:49,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:49,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:49,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:49,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:49,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:49,747 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:53:49,748 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f7512a5[Count = 0] remaining members to acquire global barrier 2023-03-31 16:53:49,748 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:53:49,748 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:49,750 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:49,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:49,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:49,751 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:53:49,751 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:53:49,751 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:53:49,751 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:49,751 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:53:49,753 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:53:49,753 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:49,753 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:49,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:49,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:49,753 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:53:49,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:49,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:49,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:49,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:49,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:49,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:49,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:49,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:49,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:53:49,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:53:49,761 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:53:49,761 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:53:49,762 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:53:49,762 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:53:49,762 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:53:49,765 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:49,765 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:49,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:49,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:49,765 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:49,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:49,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:49,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:49,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:49,766 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:49,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:49,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:49,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:49,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:49,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:49,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:49,769 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:49,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:49,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:49,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:49,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:49,776 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:49,776 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:49,776 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:49,776 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:49,776 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:49,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:53:49,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:53:49,776 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:49,776 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:49,781 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:49,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:49,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:49,781 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:53:49,781 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:53:49,781 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:49,781 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:49,781 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:53:49,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:49,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:49,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on e94df50bb46b68756c6a704332e1c5f4 2023-03-31 16:53:49,863 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e94df50bb46b68756c6a704332e1c5f4 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-31 16:53:49,865 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=A 2023-03-31 16:53:49,866 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:53:49,866 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=B 2023-03-31 16:53:49,866 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:53:49,866 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=C 2023-03-31 16:53:49,866 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:53:49,881 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:53:49,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:53:49,886 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:53:49,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:53:49,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:53:49,895 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:53:49,899 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:49,904 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:53:49,904 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:53:49,905 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:49,905 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:49,910 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:49,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:49,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:49,911 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:49,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:49,911 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:49,911 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:53:49,912 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:49,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:53:49,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:49,912 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:53:49,913 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/A/fa6db4443ac7493a98563caf2a8e7b40 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/fa6db4443ac7493a98563caf2a8e7b40 2023-03-31 16:53:49,922 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:53:49,922 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:53:49,922 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:53:49,922 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:49,922 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=148 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/A/68827ed2391e4ca29a272cb933a5cedd 2023-03-31 16:53:49,923 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:53:49,924 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:53:49,924 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:53:49,924 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:49,924 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. started... 2023-03-31 16:53:49,924 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. as already flushing 2023-03-31 16:53:49,924 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:49,924 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:53:49,924 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:53:49,924 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:49,924 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:53:49,925 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:53:49,930 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:49,930 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:49,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:49,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:49,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:49,931 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:49,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:49,931 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:53:49,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:49,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:49,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:49,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:49,934 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:53:49,934 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3463763c[Count = 0] remaining members to acquire global barrier 2023-03-31 16:53:49,934 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:53:49,935 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:49,936 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:49,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:49,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:49,936 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:53:49,936 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:53:49,936 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:53:49,936 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:49,936 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:53:49,938 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:49,939 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:53:49,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:49,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:49,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:49,939 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:49,939 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:53:49,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:49,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:49,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:49,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:49,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:49,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:49,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:49,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:53:49,948 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:53:49,948 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:53:49,948 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:53:49,948 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:53:49,948 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:53:49,948 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:53:49,951 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:49,952 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:49,952 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:49,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:49,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:49,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:49,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:49,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:49,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:49,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:49,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:49,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:49,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:49,954 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:49,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:49,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:49,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:49,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:49,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:49,957 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:49,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:49,970 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:49,970 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:49,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:49,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:49,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:49,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:49,971 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:49,971 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:49,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:53:49,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:53:49,972 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:49,972 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:49,972 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:49,972 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:49,972 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:49,971 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:49,972 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:53:49,972 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:53:49,979 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=148 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/B/3fc5cb40778f42ceab1413af62d9ea65 2023-03-31 16:53:49,980 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:49,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54630 deadline: 1680281689903, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:49,981 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1680281629892, max timestamp=1680281629919Num uniques 859; 2023-03-31 16:53:49,981 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-31 16:53:49,980 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:49,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54628 deadline: 1680281689901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:49,981 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in e94df50bb46b68756c6a704332e1c5f4/A of e94df50bb46b68756c6a704332e1c5f4 into fa6db4443ac7493a98563caf2a8e7b40(size=11.8 K), total size for store is 23.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:53:49,982 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e94df50bb46b68756c6a704332e1c5f4: 2023-03-31 16:53:49,982 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4., storeName=e94df50bb46b68756c6a704332e1c5f4/A, priority=11, startTime=1680281629358; duration=0sec 2023-03-31 16:53:49,982 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:53:49,982 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680281629892, max timestamp=1680281629919Num uniques 859; 2023-03-31 16:53:49,982 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-03-31 16:53:49,982 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680281629892, max timestamp=1680281629919Num uniques 859; 2023-03-31 16:53:49,982 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-03-31 16:53:50,035 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:50,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54604 deadline: 1680281690023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,040 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:50,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54638 deadline: 1680281690034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,041 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:50,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54602 deadline: 1680281690036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,072 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:53:50,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:53:50,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:53:50,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:53:50,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:53:50,078 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:53:50,079 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:50,081 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:53:50,081 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:53:50,082 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:50,082 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:50,084 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:50,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:50,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:50,084 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:50,085 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,085 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:53:50,085 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:50,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:53:50,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:50,086 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:53:50,086 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:50,086 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:53:50,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54630 deadline: 1680281690083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,086 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:53:50,086 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:53:50,086 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:50,089 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:53:50,089 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:53:50,089 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:50,089 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:50,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54628 deadline: 1680281690084, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,089 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:53:50,089 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. started... 2023-03-31 16:53:50,089 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. as already flushing 2023-03-31 16:53:50,089 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:50,089 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:53:50,089 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:53:50,089 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,090 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:53:50,090 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:53:50,093 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:50,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:50,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:50,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:50,094 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:50,095 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:50,095 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:53:50,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:50,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:50,097 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:53:50,097 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1742140a[Count = 0] remaining members to acquire global barrier 2023-03-31 16:53:50,097 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:53:50,097 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:50,098 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:50,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:50,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:50,098 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:53:50,098 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:53:50,098 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,099 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:53:50,098 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:53:50,100 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:53:50,100 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:50,100 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:53:50,101 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:50,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:50,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:50,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:50,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:50,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:50,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:50,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:53:50,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:53:50,119 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:53:50,119 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:53:50,119 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:53:50,119 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:53:50,119 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:53:50,121 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:50,121 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:50,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:50,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:50,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:50,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:50,121 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:50,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:50,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:50,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:50,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:50,122 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:50,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:50,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:50,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:50,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,124 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:50,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:50,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,128 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:50,129 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:53:50,129 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:53:50,129 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,129 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:50,129 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:50,129 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,129 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:50,129 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:50,129 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:53:50,129 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:53:50,129 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:50,129 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:50,129 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:50,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:50,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:50,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:50,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:50,141 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:50,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54604 deadline: 1680281690139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,146 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:50,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54638 deadline: 1680281690142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,146 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:50,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54602 deadline: 1680281690143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,229 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:53:50,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:53:50,233 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:53:50,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:53:50,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:53:50,244 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:53:50,244 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:50,246 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:53:50,246 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:53:50,247 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:50,247 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:50,249 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:50,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:50,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:50,249 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:50,250 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:50,250 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,250 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:53:50,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:53:50,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:50,251 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:53:50,256 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:53:50,256 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:53:50,256 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:53:50,256 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:50,258 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:53:50,259 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:53:50,259 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:53:50,259 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:50,259 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. started... 2023-03-31 16:53:50,259 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. as already flushing 2023-03-31 16:53:50,259 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:50,260 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:53:50,260 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:53:50,260 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,260 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:53:50,260 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:53:50,262 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:50,262 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:50,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:50,262 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:50,262 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:53:50,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:50,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:50,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:50,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:50,269 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:53:50,269 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@212139bc[Count = 0] remaining members to acquire global barrier 2023-03-31 16:53:50,269 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:53:50,269 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:50,271 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:50,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:50,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:50,272 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:53:50,272 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:53:50,272 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:53:50,272 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,272 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:53:50,274 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:50,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:50,275 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:53:50,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:50,275 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:50,275 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:53:50,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:50,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:50,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:50,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:50,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:53:50,285 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:53:50,285 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:53:50,285 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:53:50,285 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:53:50,285 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:53:50,285 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:53:50,287 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:50,287 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:50,287 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:50,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:50,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:50,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:50,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:50,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:50,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:50,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:50,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:50,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:50,288 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:50,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:50,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:50,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:50,293 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:50,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,297 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,297 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:50,297 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:50,297 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:50,298 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:50,298 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:50,298 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:53:50,298 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:53:50,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:50,298 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:50,298 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:50,298 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:50,301 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:53:50,301 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:50,301 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:53:50,302 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:50,302 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:50,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54630 deadline: 1680281690293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54628 deadline: 1680281690293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:50,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:50,349 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:50,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54604 deadline: 1680281690344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,353 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:50,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54638 deadline: 1680281690348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,353 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:50,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54602 deadline: 1680281690349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,401 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:53:50,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:53:50,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:53:50,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:53:50,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:53:50,412 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:53:50,412 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:50,413 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:53:50,413 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:53:50,414 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:50,414 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:50,417 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:50,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:50,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:50,417 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:50,418 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,418 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:53:50,418 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:50,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:53:50,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:50,419 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:53:50,432 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:53:50,432 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:53:50,432 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:53:50,432 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:50,437 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:53:50,437 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:53:50,437 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:53:50,437 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:50,437 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. started... 2023-03-31 16:53:50,438 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. as already flushing 2023-03-31 16:53:50,438 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:50,438 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:53:50,438 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:53:50,438 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,438 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:53:50,438 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:53:50,439 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=148 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/C/7474ae0d11f3466786c35e1681db04fb 2023-03-31 16:53:50,440 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:50,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:50,441 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:50,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:50,443 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:50,443 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:53:50,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:50,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:50,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:50,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:53:50,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2646f707[Count = 0] remaining members to acquire global barrier 2023-03-31 16:53:50,448 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:53:50,449 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:50,453 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:50,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:50,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:50,453 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,453 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:53:50,453 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:53:50,453 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:53:50,453 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:53:50,454 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/A/68827ed2391e4ca29a272cb933a5cedd as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/68827ed2391e4ca29a272cb933a5cedd 2023-03-31 16:53:50,458 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:50,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:50,460 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:53:50,460 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:50,460 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:53:50,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:50,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:50,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:50,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:50,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:50,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,472 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/68827ed2391e4ca29a272cb933a5cedd, entries=150, sequenceid=148, filesize=11.8 K 2023-03-31 16:53:50,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:53:50,474 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:53:50,474 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:53:50,474 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:53:50,474 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:53:50,474 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:53:50,474 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:53:50,476 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/B/3fc5cb40778f42ceab1413af62d9ea65 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/3fc5cb40778f42ceab1413af62d9ea65 2023-03-31 16:53:50,481 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:50,481 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:50,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:50,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:50,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:50,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:50,481 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:50,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:50,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:50,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:50,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:50,483 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:50,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:50,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:50,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:50,486 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:50,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:50,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,489 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:50,489 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:50,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:50,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:50,489 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,489 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:50,490 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:50,490 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:50,490 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:53:50,493 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:53:50,493 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,493 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:50,493 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:50,494 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:50,493 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:53:50,494 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:53:50,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:50,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:50,501 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/3fc5cb40778f42ceab1413af62d9ea65, entries=150, sequenceid=148, filesize=11.8 K 2023-03-31 16:53:50,510 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/C/7474ae0d11f3466786c35e1681db04fb as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/7474ae0d11f3466786c35e1681db04fb 2023-03-31 16:53:50,539 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/7474ae0d11f3466786c35e1681db04fb, entries=150, sequenceid=148, filesize=11.8 K 2023-03-31 16:53:50,542 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=117.74 KB/120562 for e94df50bb46b68756c6a704332e1c5f4 in 679ms, sequenceid=148, compaction requested=true 2023-03-31 16:53:50,542 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e94df50bb46b68756c6a704332e1c5f4: 2023-03-31 16:53:50,543 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:53:50,543 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 16:53:50,543 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 16:53:50,543 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:53:50,543 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:53:50,549 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36130 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 16:53:50,549 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36130 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 16:53:50,549 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): e94df50bb46b68756c6a704332e1c5f4/B is initiating minor compaction (all files) 2023-03-31 16:53:50,549 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1912): e94df50bb46b68756c6a704332e1c5f4/A is initiating minor compaction (all files) 2023-03-31 16:53:50,549 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e94df50bb46b68756c6a704332e1c5f4/B in TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:50,549 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e94df50bb46b68756c6a704332e1c5f4/A in TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:50,549 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/1f1c599c73c840b584187ea68eb50fc4, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/db2a8599b6cb4821bce4873cfd83b9f5, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/3fc5cb40778f42ceab1413af62d9ea65] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp, totalSize=35.3 K 2023-03-31 16:53:50,549 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/fa6db4443ac7493a98563caf2a8e7b40, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/c447ed0d8629424e93db831f3698ec0e, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/68827ed2391e4ca29a272cb933a5cedd] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp, totalSize=35.3 K 2023-03-31 16:53:50,550 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 1f1c599c73c840b584187ea68eb50fc4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1680281628988 2023-03-31 16:53:50,551 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting fa6db4443ac7493a98563caf2a8e7b40, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1680281628988 2023-03-31 16:53:50,552 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting c447ed0d8629424e93db831f3698ec0e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=133, earliestPutTs=1680281629127 2023-03-31 16:53:50,552 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting db2a8599b6cb4821bce4873cfd83b9f5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=133, earliestPutTs=1680281629127 2023-03-31 16:53:50,553 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 68827ed2391e4ca29a272cb933a5cedd, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1680281629506 2023-03-31 16:53:50,553 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 3fc5cb40778f42ceab1413af62d9ea65, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1680281629506 2023-03-31 16:53:50,574 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] throttle.PressureAwareThroughputController(145): e94df50bb46b68756c6a704332e1c5f4#A#compaction#24 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:53:50,590 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): e94df50bb46b68756c6a704332e1c5f4#B#compaction#25 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:53:50,594 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:53:50,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:53:50,608 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:53:50,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on e94df50bb46b68756c6a704332e1c5f4 2023-03-31 16:53:50,615 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e94df50bb46b68756c6a704332e1c5f4 3/3 column families, dataSize=124.45 KB heapSize=353.47 KB 2023-03-31 16:53:50,622 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=A 2023-03-31 16:53:50,622 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:53:50,622 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=B 2023-03-31 16:53:50,622 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:53:50,622 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=C 2023-03-31 16:53:50,622 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:53:50,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:53:50,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:53:50,629 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:53:50,629 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:50,631 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:53:50,631 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:53:50,632 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:50,632 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:50,634 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,634 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:50,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:50,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:50,635 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,635 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:53:50,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:50,644 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/A/07320ee76d544ccea3a912d574b89806 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/07320ee76d544ccea3a912d574b89806 2023-03-31 16:53:50,644 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:50,644 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/B/921c9b07d4ac47cea568c5511a5fdcc7 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/921c9b07d4ac47cea568c5511a5fdcc7 2023-03-31 16:53:50,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:53:50,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:50,645 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:53:50,654 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:53:50,654 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:53:50,655 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:53:50,655 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:50,657 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e94df50bb46b68756c6a704332e1c5f4/B of e94df50bb46b68756c6a704332e1c5f4 into 921c9b07d4ac47cea568c5511a5fdcc7(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:53:50,657 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e94df50bb46b68756c6a704332e1c5f4: 2023-03-31 16:53:50,657 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4., storeName=e94df50bb46b68756c6a704332e1c5f4/B, priority=13, startTime=1680281630543; duration=0sec 2023-03-31 16:53:50,657 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:53:50,657 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 16:53:50,658 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e94df50bb46b68756c6a704332e1c5f4/A of e94df50bb46b68756c6a704332e1c5f4 into 07320ee76d544ccea3a912d574b89806(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:53:50,658 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e94df50bb46b68756c6a704332e1c5f4: 2023-03-31 16:53:50,658 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4., storeName=e94df50bb46b68756c6a704332e1c5f4/A, priority=13, startTime=1680281630543; duration=0sec 2023-03-31 16:53:50,659 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:53:50,660 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:53:50,660 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36130 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 16:53:50,661 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): e94df50bb46b68756c6a704332e1c5f4/C is initiating minor compaction (all files) 2023-03-31 16:53:50,662 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e94df50bb46b68756c6a704332e1c5f4/C in TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:50,663 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/1ce9b06377fe4597b18ab6b3bcf0ee7c, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/8da43654ffe04f1f9168f9c43e5e3263, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/7474ae0d11f3466786c35e1681db04fb] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp, totalSize=35.3 K 2023-03-31 16:53:50,667 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 1ce9b06377fe4597b18ab6b3bcf0ee7c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1680281628988 2023-03-31 16:53:50,669 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:53:50,671 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:53:50,669 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:50,671 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 8da43654ffe04f1f9168f9c43e5e3263, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=133, earliestPutTs=1680281629127 2023-03-31 16:53:50,671 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. started... 2023-03-31 16:53:50,672 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. as already flushing 2023-03-31 16:53:50,672 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:50,672 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:53:50,672 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:53:50,672 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,672 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:53:50,672 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:53:50,673 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 7474ae0d11f3466786c35e1681db04fb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1680281629506 2023-03-31 16:53:50,676 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=185 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/A/ca36732fc7d04a5caee63a13f295f814 2023-03-31 16:53:50,679 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:50,679 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:50,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:50,680 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:50,680 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:53:50,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:50,681 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:50,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:50,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54638 deadline: 1680281690657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,681 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:50,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54602 deadline: 1680281690657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,682 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:50,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:50,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54630 deadline: 1680281690660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,682 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:50,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54604 deadline: 1680281690669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,688 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:50,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54628 deadline: 1680281690681, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:50,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:53:50,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9bbb765[Count = 0] remaining members to acquire global barrier 2023-03-31 16:53:50,689 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:53:50,689 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:50,690 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:50,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:50,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:50,690 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:53:50,691 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:53:50,691 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:53:50,691 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,691 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:53:50,692 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:50,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:50,693 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:53:50,693 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:50,693 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:53:50,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:50,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:50,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:50,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:50,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:50,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:53:50,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:53:50,704 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:53:50,704 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:53:50,704 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:53:50,704 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:53:50,704 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:53:50,709 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): e94df50bb46b68756c6a704332e1c5f4#C#compaction#28 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:53:50,709 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:50,709 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:50,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:50,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:50,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:50,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:50,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:50,709 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:50,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:50,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:50,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:50,721 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,722 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.10 KB at sequenceid=185 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/B/4243b53d2d404612b39aa896086ca145 2023-03-31 16:53:50,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:50,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:50,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:50,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:50,726 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:50,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:50,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,734 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:50,734 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:50,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:50,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:50,734 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,734 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:50,734 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:50,734 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,734 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:50,734 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:50,734 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:50,734 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:50,734 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:53:50,734 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:53:50,734 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:53:50,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:50,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:50,735 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:53:50,735 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:53:50,758 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.10 KB at sequenceid=185 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/C/4ca81fd821514dd0833a312accbe5210 2023-03-31 16:53:50,764 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/C/cccccad0b7854e07bf1f553e86f4675e as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/cccccad0b7854e07bf1f553e86f4675e 2023-03-31 16:53:50,771 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/A/ca36732fc7d04a5caee63a13f295f814 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/ca36732fc7d04a5caee63a13f295f814 2023-03-31 16:53:50,774 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e94df50bb46b68756c6a704332e1c5f4/C of e94df50bb46b68756c6a704332e1c5f4 into cccccad0b7854e07bf1f553e86f4675e(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:53:50,774 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e94df50bb46b68756c6a704332e1c5f4: 2023-03-31 16:53:50,774 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4., storeName=e94df50bb46b68756c6a704332e1c5f4/C, priority=13, startTime=1680281630543; duration=0sec 2023-03-31 16:53:50,774 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:53:50,783 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/ca36732fc7d04a5caee63a13f295f814, entries=250, sequenceid=185, filesize=16.4 K 2023-03-31 16:53:50,792 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/B/4243b53d2d404612b39aa896086ca145 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/4243b53d2d404612b39aa896086ca145 2023-03-31 16:53:50,798 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:50,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54604 deadline: 1680281690784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,799 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:50,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54630 deadline: 1680281690793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,808 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:50,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54628 deadline: 1680281690798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,832 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/4243b53d2d404612b39aa896086ca145, entries=150, sequenceid=185, filesize=11.8 K 2023-03-31 16:53:50,835 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/C/4ca81fd821514dd0833a312accbe5210 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/4ca81fd821514dd0833a312accbe5210 2023-03-31 16:53:50,835 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:53:50,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:53:50,844 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:53:50,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:53:50,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:53:50,849 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:53:50,849 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:50,853 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:53:50,853 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:53:50,854 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:50,854 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:50,856 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/4ca81fd821514dd0833a312accbe5210, entries=150, sequenceid=185, filesize=11.8 K 2023-03-31 16:53:50,859 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,859 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:50,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:50,859 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:50,859 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:53:50,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:50,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:50,860 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:50,860 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~131.15 KB/134302, heapSize ~373.34 KB/382304, currentSize=46.96 KB/48090 for e94df50bb46b68756c6a704332e1c5f4 in 245ms, sequenceid=185, compaction requested=false 2023-03-31 16:53:50,861 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e94df50bb46b68756c6a704332e1c5f4: 2023-03-31 16:53:50,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:53:50,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:50,861 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:53:50,862 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:53:50,863 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:53:50,863 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:53:50,863 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:50,865 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:53:50,865 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:53:50,865 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:53:50,865 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:50,865 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. started... 2023-03-31 16:53:50,865 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e94df50bb46b68756c6a704332e1c5f4 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-31 16:53:50,866 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=A 2023-03-31 16:53:50,866 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:53:50,866 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=B 2023-03-31 16:53:50,866 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:53:50,866 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=C 2023-03-31 16:53:50,866 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:53:50,886 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=198 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/A/d61f5d3191e243a481f2b953ec45220a 2023-03-31 16:53:51,004 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. as already flushing 2023-03-31 16:53:51,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on e94df50bb46b68756c6a704332e1c5f4 2023-03-31 16:53:51,041 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680281631024, max timestamp=1680281631037Num uniques 859; 2023-03-31 16:53:51,041 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 16:53:51,041 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680281631024, max timestamp=1680281631037Num uniques 859; 2023-03-31 16:53:51,041 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 16:53:51,041 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680281631024, max timestamp=1680281631037Num uniques 859; 2023-03-31 16:53:51,042 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 16:53:51,096 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:51,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54630 deadline: 1680281691090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:51,096 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:51,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54628 deadline: 1680281691091, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:51,097 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:51,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54604 deadline: 1680281691093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:51,187 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:51,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54638 deadline: 1680281691184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:51,187 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:51,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54602 deadline: 1680281691187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:51,199 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:51,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54630 deadline: 1680281691198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:51,200 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:51,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54628 deadline: 1680281691199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:51,201 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:51,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54604 deadline: 1680281691199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:51,341 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=198 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/B/ee77cc8f635944a3a75c3485003050e5 2023-03-31 16:53:51,390 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=198 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/C/14ecae34094a42c29e8592284aeeb44f 2023-03-31 16:53:51,400 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/A/d61f5d3191e243a481f2b953ec45220a as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/d61f5d3191e243a481f2b953ec45220a 2023-03-31 16:53:51,410 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:51,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54604 deadline: 1680281691403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:51,413 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:51,414 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:51,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54630 deadline: 1680281691402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:51,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54628 deadline: 1680281691403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:51,415 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/d61f5d3191e243a481f2b953ec45220a, entries=150, sequenceid=198, filesize=11.8 K 2023-03-31 16:53:51,420 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/B/ee77cc8f635944a3a75c3485003050e5 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/ee77cc8f635944a3a75c3485003050e5 2023-03-31 16:53:51,429 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/ee77cc8f635944a3a75c3485003050e5, entries=150, sequenceid=198, filesize=11.8 K 2023-03-31 16:53:51,432 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/C/14ecae34094a42c29e8592284aeeb44f as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/14ecae34094a42c29e8592284aeeb44f 2023-03-31 16:53:51,439 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/14ecae34094a42c29e8592284aeeb44f, entries=150, sequenceid=198, filesize=11.8 K 2023-03-31 16:53:51,441 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for e94df50bb46b68756c6a704332e1c5f4 in 576ms, sequenceid=198, compaction requested=true 2023-03-31 16:53:51,442 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e94df50bb46b68756c6a704332e1c5f4: 2023-03-31 16:53:51,442 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:51,442 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:53:51,445 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:53:51,445 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:51,447 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:53:51,448 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:53:51,451 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:51,451 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:51,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:51,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:51,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:51,451 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:51,451 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:53:51,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:51,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:51,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:51,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:51,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:51,453 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:53:51,453 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ede101c[Count = 0] remaining members to acquire global barrier 2023-03-31 16:53:51,453 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:53:51,453 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:51,455 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:51,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:51,455 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:51,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:51,455 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:53:51,455 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:53:51,456 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:53:51,456 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:53:51,460 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:51,460 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:53:51,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:51,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:51,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:51,460 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:51,461 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:53:51,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:51,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:51,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:51,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:51,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:51,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:51,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:51,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:53:51,472 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:53:51,472 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:53:51,472 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:53:51,472 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:53:51,472 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:53:51,472 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:53:51,474 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:51,475 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:51,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:51,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:51,475 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:51,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:51,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:51,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:51,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:51,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:51,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:51,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:51,476 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:51,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:51,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:51,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:51,479 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:51,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:51,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:51,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:51,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:51,485 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:51,485 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:51,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:51,485 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:51,485 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:51,485 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:51,489 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:51,485 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:53:51,490 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:53:51,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:51,490 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:51,490 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:51,490 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:53:51,490 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:51,490 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:53:51,490 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:51,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:51,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:51,591 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:53:51,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:53:51,594 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:53:51,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:53:51,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:53:51,600 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:53:51,600 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:51,601 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:53:51,601 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:53:51,602 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:51,602 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:51,603 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:51,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:51,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:51,604 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:51,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:51,604 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:51,605 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:51,605 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:53:51,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:53:51,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:51,605 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:53:51,605 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:53:51,606 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:53:51,606 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:53:51,606 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:51,607 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:53:51,607 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:53:51,607 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:53:51,608 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:51,608 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. started... 2023-03-31 16:53:51,608 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e94df50bb46b68756c6a704332e1c5f4 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-03-31 16:53:51,608 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=A 2023-03-31 16:53:51,608 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:53:51,608 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=B 2023-03-31 16:53:51,608 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:53:51,609 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=C 2023-03-31 16:53:51,609 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:53:51,630 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=235 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/A/81256392dfb144279664831b3777c42e 2023-03-31 16:53:51,660 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=235 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/B/5176ef7353b34fdd92495e699da94346 2023-03-31 16:53:51,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on e94df50bb46b68756c6a704332e1c5f4 2023-03-31 16:53:51,718 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. as already flushing 2023-03-31 16:53:51,729 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:51,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54628 deadline: 1680281691724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:51,731 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:51,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54604 deadline: 1680281691728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:51,732 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:51,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54630 deadline: 1680281691730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:51,832 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:51,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54628 deadline: 1680281691831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:51,836 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:51,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54604 deadline: 1680281691833, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:51,837 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:51,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54630 deadline: 1680281691834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,036 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:52,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54628 deadline: 1680281692035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,041 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:52,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54630 deadline: 1680281692038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,044 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:52,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54604 deadline: 1680281692039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,097 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=235 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/C/bd6ffc8b9dfc45728b59def07fe6e912 2023-03-31 16:53:52,105 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/A/81256392dfb144279664831b3777c42e as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/81256392dfb144279664831b3777c42e 2023-03-31 16:53:52,116 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/81256392dfb144279664831b3777c42e, entries=150, sequenceid=235, filesize=11.8 K 2023-03-31 16:53:52,119 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/B/5176ef7353b34fdd92495e699da94346 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/5176ef7353b34fdd92495e699da94346 2023-03-31 16:53:52,126 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/5176ef7353b34fdd92495e699da94346, entries=150, sequenceid=235, filesize=11.8 K 2023-03-31 16:53:52,130 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/C/bd6ffc8b9dfc45728b59def07fe6e912 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/bd6ffc8b9dfc45728b59def07fe6e912 2023-03-31 16:53:52,139 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/bd6ffc8b9dfc45728b59def07fe6e912, entries=150, sequenceid=235, filesize=11.8 K 2023-03-31 16:53:52,140 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=40.25 KB/41220 for e94df50bb46b68756c6a704332e1c5f4 in 532ms, sequenceid=235, compaction requested=true 2023-03-31 16:53:52,141 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e94df50bb46b68756c6a704332e1c5f4: 2023-03-31 16:53:52,141 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:52,141 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:53:52,142 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:53:52,142 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,142 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:53:52,142 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:53:52,145 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,145 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:52,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:52,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:52,145 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:52,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:52,146 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:53:52,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:52,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:52,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:52,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:53:52,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@269a34a2[Count = 0] remaining members to acquire global barrier 2023-03-31 16:53:52,147 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:53:52,147 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:52,149 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:52,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:52,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:52,149 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:53:52,149 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:53:52,149 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:53:52,149 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,149 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:53:52,150 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:53:52,150 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,150 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:52,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,151 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:53:52,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:52,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:52,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:52,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:52,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:52,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:52,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:52,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:53:52,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:53:52,160 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:53:52,161 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:53:52,161 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:53:52,161 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:53:52,161 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:53:52,167 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:52,167 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:52,168 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:52,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:52,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:52,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:52,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:52,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:52,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:52,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:52,169 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:52,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:52,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:52,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:52,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:52,172 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:52,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:52,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,175 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:52,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:52,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:52,175 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:52,175 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,175 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:52,175 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:52,175 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:52,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:52,175 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:53:52,176 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:53:52,175 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:52,175 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:52,176 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:53:52,177 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:53:52,176 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:52,177 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:52,176 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:53:52,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on e94df50bb46b68756c6a704332e1c5f4 2023-03-31 16:53:52,193 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e94df50bb46b68756c6a704332e1c5f4 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-31 16:53:52,193 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=A 2023-03-31 16:53:52,193 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:53:52,194 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=B 2023-03-31 16:53:52,194 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:53:52,194 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=C 2023-03-31 16:53:52,194 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:53:52,208 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=245 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/A/81fd752e0ce5425ba2e00f25e99baeb0 2023-03-31 16:53:52,242 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680281632228, max timestamp=1680281632240Num uniques 859; 2023-03-31 16:53:52,242 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-31 16:53:52,242 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680281632228, max timestamp=1680281632240Num uniques 859; 2023-03-31 16:53:52,242 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680281632228, max timestamp=1680281632240Num uniques 859; 2023-03-31 16:53:52,242 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-31 16:53:52,242 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-31 16:53:52,277 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:52,277 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:53:52,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54602 deadline: 1680281692276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:53:52,280 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:53:52,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:53:52,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:53:52,284 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:53:52,284 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:52,285 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:53:52,285 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:53:52,286 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:52,286 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:52,286 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:52,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54638 deadline: 1680281692276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,287 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:52,287 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:52,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:52,288 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,288 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:53:52,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:52,288 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:52,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:53:52,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:52,289 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:53:52,289 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:53:52,289 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:53:52,289 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:53:52,289 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:52,290 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:53:52,290 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:53:52,290 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:53:52,291 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:52,291 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. started... 2023-03-31 16:53:52,291 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. as already flushing 2023-03-31 16:53:52,291 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:52,291 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:53:52,291 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:53:52,291 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,291 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:53:52,291 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:53:52,292 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:52,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:52,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:52,293 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:52,294 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:52,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:52,294 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:53:52,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:52,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:52,296 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:53:52,296 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@377501b9[Count = 0] remaining members to acquire global barrier 2023-03-31 16:53:52,296 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:53:52,296 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:52,298 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:52,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:52,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:52,298 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:53:52,298 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:53:52,298 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:53:52,302 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,302 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:53:52,303 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:53:52,303 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,303 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:52,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:52,303 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:53:52,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:52,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:52,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:52,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:52,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:52,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:52,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:53:52,314 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:53:52,314 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:53:52,314 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:53:52,314 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:53:52,314 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:53:52,314 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:53:52,317 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:52,318 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:52,318 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:52,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:52,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:52,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:52,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:52,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:52,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:52,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:52,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:52,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:52,320 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:52,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:52,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:52,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,322 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:52,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:52,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,326 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,326 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:52,326 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:52,326 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:52,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:52,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:52,326 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:52,326 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,327 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:52,327 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:52,327 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:52,327 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:52,327 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:53:52,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:53:52,328 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:53:52,328 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:53:52,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:52,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:52,342 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:52,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54628 deadline: 1680281692339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,347 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:52,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54630 deadline: 1680281692345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,351 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:52,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54604 deadline: 1680281692348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,382 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:52,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54602 deadline: 1680281692379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,393 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:52,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54638 deadline: 1680281692388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,428 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:53:52,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:53:52,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:53:52,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:53:52,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:53:52,448 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:53:52,448 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:52,449 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:53:52,449 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:53:52,450 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:52,450 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:52,453 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:52,453 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:52,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:52,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:52,454 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,454 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:53:52,454 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:52,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:53:52,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:52,454 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:53:52,455 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:53:52,455 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:53:52,455 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:53:52,455 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:52,458 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:53:52,458 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:53:52,459 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:53:52,459 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:52,459 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. started... 2023-03-31 16:53:52,459 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. as already flushing 2023-03-31 16:53:52,459 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:52,459 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:53:52,459 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:53:52,459 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,459 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:53:52,459 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:53:52,460 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,460 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:52,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:52,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:52,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:52,461 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:52,461 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:53:52,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:52,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:52,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:52,462 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:53:52,462 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63a198ad[Count = 0] remaining members to acquire global barrier 2023-03-31 16:53:52,462 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:53:52,462 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:52,464 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:52,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:52,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:52,464 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:53:52,464 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:53:52,464 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:53:52,464 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,464 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:53:52,466 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:53:52,466 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:52,466 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:53:52,467 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:52,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:52,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:52,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:52,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:52,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:52,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:52,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:53:52,480 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:53:52,480 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:53:52,480 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:53:52,481 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:53:52,481 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:53:52,481 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:53:52,484 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:52,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:52,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:52,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:52,485 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:52,485 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:52,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:52,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:52,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:52,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:52,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:52,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:52,486 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:52,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:52,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:52,489 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:52,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:52,493 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:52,493 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:52,494 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:52,494 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:52,494 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,494 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:52,494 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:52,494 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:52,494 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:52,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:52,497 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:52,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:53:52,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:53:52,511 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:53:52,511 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:53:52,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:52,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:52,515 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:53:52,592 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:52,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54602 deadline: 1680281692585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,598 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:52,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54638 deadline: 1680281692596, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,611 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:53:52,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:53:52,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:53:52,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:53:52,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:53:52,619 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:53:52,619 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:52,619 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:53:52,619 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:53:52,619 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:52,619 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:52,621 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:52,621 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:52,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:52,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:52,628 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,628 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:52,628 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:53:52,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:53:52,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:52,632 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:53:52,644 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:53:52,644 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:53:52,644 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=245 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/B/d6a8d304ad1f4643918d10cc12956c6a 2023-03-31 16:53:52,644 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:53:52,644 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:52,644 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:53:52,656 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:53:52,656 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:53:52,657 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:52,657 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. started... 2023-03-31 16:53:52,657 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. as already flushing 2023-03-31 16:53:52,657 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:52,657 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:53:52,657 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:53:52,657 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,673 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:53:52,674 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:53:52,676 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,677 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:52,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:52,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:52,678 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:52,678 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:53:52,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:52,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:52,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:52,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:52,683 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:53:52,683 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d812a35[Count = 0] remaining members to acquire global barrier 2023-03-31 16:53:52,683 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:53:52,683 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:52,685 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:52,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:52,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:52,685 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:53:52,685 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:53:52,685 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:53:52,685 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,685 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:53:52,689 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,689 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:53:52,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:52,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:52,689 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:52,689 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:53:52,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:52,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:52,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:52,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:52,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:52,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:53:52,693 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:53:52,693 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:53:52,693 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:53:52,693 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:53:52,693 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:53:52,693 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:53:52,695 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:52,695 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:52,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:52,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:52,696 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:52,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:52,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:52,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:52,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:52,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:52,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:52,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:52,699 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:52,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:52,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:52,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,704 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:52,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:52,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,712 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:52,712 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,712 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:52,712 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:52,712 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:52,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:52,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:52,712 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,712 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:52,713 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:52,713 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:52,713 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:53:52,713 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:53:52,713 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:52,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:52,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:52,720 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:53:52,720 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:53:52,722 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=245 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/C/22a9c62af3ae44cfb233da38fb3f0b44 2023-03-31 16:53:52,732 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/A/81fd752e0ce5425ba2e00f25e99baeb0 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/81fd752e0ce5425ba2e00f25e99baeb0 2023-03-31 16:53:52,742 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/81fd752e0ce5425ba2e00f25e99baeb0, entries=200, sequenceid=245, filesize=14.1 K 2023-03-31 16:53:52,744 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/B/d6a8d304ad1f4643918d10cc12956c6a as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/d6a8d304ad1f4643918d10cc12956c6a 2023-03-31 16:53:52,753 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/d6a8d304ad1f4643918d10cc12956c6a, entries=150, sequenceid=245, filesize=11.8 K 2023-03-31 16:53:52,754 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/C/22a9c62af3ae44cfb233da38fb3f0b44 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/22a9c62af3ae44cfb233da38fb3f0b44 2023-03-31 16:53:52,782 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/22a9c62af3ae44cfb233da38fb3f0b44, entries=150, sequenceid=245, filesize=11.8 K 2023-03-31 16:53:52,783 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for e94df50bb46b68756c6a704332e1c5f4 in 590ms, sequenceid=245, compaction requested=true 2023-03-31 16:53:52,784 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e94df50bb46b68756c6a704332e1c5f4: 2023-03-31 16:53:52,784 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:53:52,784 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:53:52,784 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-31 16:53:52,784 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-31 16:53:52,784 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-31 16:53:52,790 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 67624 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-31 16:53:52,790 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60454 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-31 16:53:52,790 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1912): e94df50bb46b68756c6a704332e1c5f4/A is initiating minor compaction (all files) 2023-03-31 16:53:52,791 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): e94df50bb46b68756c6a704332e1c5f4/B is initiating minor compaction (all files) 2023-03-31 16:53:52,791 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e94df50bb46b68756c6a704332e1c5f4/A in TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:52,791 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e94df50bb46b68756c6a704332e1c5f4/B in TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:52,791 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/07320ee76d544ccea3a912d574b89806, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/ca36732fc7d04a5caee63a13f295f814, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/d61f5d3191e243a481f2b953ec45220a, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/81256392dfb144279664831b3777c42e, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/81fd752e0ce5425ba2e00f25e99baeb0] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp, totalSize=66.0 K 2023-03-31 16:53:52,791 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/921c9b07d4ac47cea568c5511a5fdcc7, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/4243b53d2d404612b39aa896086ca145, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/ee77cc8f635944a3a75c3485003050e5, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/5176ef7353b34fdd92495e699da94346, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/d6a8d304ad1f4643918d10cc12956c6a] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp, totalSize=59.0 K 2023-03-31 16:53:52,793 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 07320ee76d544ccea3a912d574b89806, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1680281629506 2023-03-31 16:53:52,798 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 921c9b07d4ac47cea568c5511a5fdcc7, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1680281629506 2023-03-31 16:53:52,799 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 4243b53d2d404612b39aa896086ca145, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=185, earliestPutTs=1680281630019 2023-03-31 16:53:52,800 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting ca36732fc7d04a5caee63a13f295f814, keycount=250, bloomtype=ROW, size=16.4 K, encoding=NONE, compression=NONE, seqNum=185, earliestPutTs=1680281630004 2023-03-31 16:53:52,800 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting ee77cc8f635944a3a75c3485003050e5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=198, earliestPutTs=1680281630654 2023-03-31 16:53:52,802 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting d61f5d3191e243a481f2b953ec45220a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=198, earliestPutTs=1680281630654 2023-03-31 16:53:52,802 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 5176ef7353b34fdd92495e699da94346, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=235, earliestPutTs=1680281631083 2023-03-31 16:53:52,802 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 81256392dfb144279664831b3777c42e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=235, earliestPutTs=1680281631083 2023-03-31 16:53:52,803 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting d6a8d304ad1f4643918d10cc12956c6a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=245, earliestPutTs=1680281631716 2023-03-31 16:53:52,804 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 81fd752e0ce5425ba2e00f25e99baeb0, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=245, earliestPutTs=1680281631716 2023-03-31 16:53:52,820 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:53:52,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:53:52,824 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:53:52,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:53:52,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:53:52,828 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:53:52,828 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:52,829 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:53:52,829 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:53:52,829 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:52,829 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:52,830 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,830 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:52,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:52,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:52,831 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,831 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:53:52,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:52,832 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:52,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:53:52,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:52,832 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:53:52,842 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] throttle.PressureAwareThroughputController(145): e94df50bb46b68756c6a704332e1c5f4#A#compaction#39 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:53:52,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on e94df50bb46b68756c6a704332e1c5f4 2023-03-31 16:53:52,852 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e94df50bb46b68756c6a704332e1c5f4 3/3 column families, dataSize=140.89 KB heapSize=395.53 KB 2023-03-31 16:53:52,852 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:53:52,853 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:53:52,853 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:53:52,853 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:52,853 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:53:52,853 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=A 2023-03-31 16:53:52,853 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:53:52,853 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=B 2023-03-31 16:53:52,853 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:53:52,853 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=C 2023-03-31 16:53:52,854 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:53:52,858 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): e94df50bb46b68756c6a704332e1c5f4#B#compaction#40 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:53:52,860 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:53:52,860 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:53:52,861 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:52,861 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. started... 2023-03-31 16:53:52,861 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. as already flushing 2023-03-31 16:53:52,862 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:52,862 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:53:52,862 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:53:52,862 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,869 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:52,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54628 deadline: 1680281692868, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,875 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:52,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54630 deadline: 1680281692870, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,876 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:52,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54604 deadline: 1680281692871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,894 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:53:52,894 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:53:52,897 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:52,897 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:52,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:52,897 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:52,897 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:53:52,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:52,900 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=282 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/A/f802b521ea6349c78739bcfc3a5b5b21 2023-03-31 16:53:52,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:52,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:52,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:52,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:53:52,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a492203[Count = 0] remaining members to acquire global barrier 2023-03-31 16:53:52,905 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:53:52,905 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:52,906 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:52,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:52,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:52,906 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:52,906 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:53:52,907 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:53:52,907 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:53:52,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54602 deadline: 1680281692900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,909 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:52,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54638 deadline: 1680281692904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,909 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:53:52,909 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:52,909 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:53:52,910 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,912 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:53:52,912 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:53:52,912 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:53:52,912 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:53:52,912 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:53:52,912 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:53:52,912 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:53:52,912 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/B/ad0cf48c84594732a4e59d5d88c010f4 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/ad0cf48c84594732a4e59d5d88c010f4 2023-03-31 16:53:52,916 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:52,916 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:52,916 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:52,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:52,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:52,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:52,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:52,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:52,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:52,917 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:52,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:52,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:52,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:52,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:52,922 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in e94df50bb46b68756c6a704332e1c5f4/B of e94df50bb46b68756c6a704332e1c5f4 into ad0cf48c84594732a4e59d5d88c010f4(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:53:52,922 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e94df50bb46b68756c6a704332e1c5f4: 2023-03-31 16:53:52,922 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4., storeName=e94df50bb46b68756c6a704332e1c5f4/B, priority=11, startTime=1680281632784; duration=0sec 2023-03-31 16:53:52,922 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:53:52,922 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-31 16:53:52,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:52,923 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:52,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:52,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,926 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60454 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-31 16:53:52,926 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): e94df50bb46b68756c6a704332e1c5f4/C is initiating minor compaction (all files) 2023-03-31 16:53:52,926 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e94df50bb46b68756c6a704332e1c5f4/C in TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:52,926 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/cccccad0b7854e07bf1f553e86f4675e, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/4ca81fd821514dd0833a312accbe5210, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/14ecae34094a42c29e8592284aeeb44f, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/bd6ffc8b9dfc45728b59def07fe6e912, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/22a9c62af3ae44cfb233da38fb3f0b44] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp, totalSize=59.0 K 2023-03-31 16:53:52,927 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting cccccad0b7854e07bf1f553e86f4675e, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1680281629506 2023-03-31 16:53:52,928 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,933 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:52,933 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 4ca81fd821514dd0833a312accbe5210, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=185, earliestPutTs=1680281630019 2023-03-31 16:53:52,933 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:52,933 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:52,933 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,933 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:52,933 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:53:52,933 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:53:52,933 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:52,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:52,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:52,933 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:52,934 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 14ecae34094a42c29e8592284aeeb44f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=198, earliestPutTs=1680281630654 2023-03-31 16:53:52,934 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:53:52,934 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:53:52,933 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:52,934 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:52,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:52,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:52,934 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting bd6ffc8b9dfc45728b59def07fe6e912, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=235, earliestPutTs=1680281631083 2023-03-31 16:53:52,935 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 22a9c62af3ae44cfb233da38fb3f0b44, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=245, earliestPutTs=1680281631716 2023-03-31 16:53:52,943 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=282 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/B/2c04ffc461254f4298677026794e0271 2023-03-31 16:53:52,956 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): e94df50bb46b68756c6a704332e1c5f4#C#compaction#43 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:53:52,979 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:52,980 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/C/c87ddfb34b7f4b49a8a29c3211d6972a as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/c87ddfb34b7f4b49a8a29c3211d6972a 2023-03-31 16:53:52,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54604 deadline: 1680281692978, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,980 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:52,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54630 deadline: 1680281692979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,986 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:52,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54628 deadline: 1680281692984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:52,990 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in e94df50bb46b68756c6a704332e1c5f4/C of e94df50bb46b68756c6a704332e1c5f4 into c87ddfb34b7f4b49a8a29c3211d6972a(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:53:52,990 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e94df50bb46b68756c6a704332e1c5f4: 2023-03-31 16:53:52,990 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4., storeName=e94df50bb46b68756c6a704332e1c5f4/C, priority=11, startTime=1680281632784; duration=0sec 2023-03-31 16:53:52,990 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:53:53,034 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:53:53,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:53:53,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:53:53,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:53:53,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:53:53,039 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:53:53,039 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:53,039 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:53:53,039 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:53:53,040 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:53,040 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:53,041 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:53,041 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:53,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:53,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:53,042 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:53,043 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,043 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:53:53,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:53:53,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:53,043 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:53:53,043 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:53:53,043 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:53:53,043 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:53:53,044 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:53,044 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:53:53,044 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:53:53,044 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:53:53,044 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:53,044 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. started... 2023-03-31 16:53:53,044 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. as already flushing 2023-03-31 16:53:53,044 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:53,044 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:53:53,044 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:53:53,044 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,044 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:53:53,044 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:53:53,047 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,047 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:53,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:53,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:53,048 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:53,048 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:53:53,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:53,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:53,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:53,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:53,056 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:53:53,056 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:53:53,056 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75ae9e3a[Count = 0] remaining members to acquire global barrier 2023-03-31 16:53:53,056 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:53,058 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:53,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:53,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:53,058 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:53:53,059 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:53:53,059 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:53:53,059 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,060 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:53:53,061 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,061 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:53:53,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:53,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:53,061 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:53,061 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:53:53,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:53,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:53,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:53,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:53,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:53,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:53:53,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:53:53,065 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:53:53,066 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:53:53,066 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:53:53,066 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:53:53,066 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:53:53,068 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:53,068 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:53,068 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:53,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:53,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:53,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:53,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:53,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:53,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:53,069 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:53,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:53,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:53,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:53,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:53,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:53,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,071 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:53,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:53,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,074 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,074 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:53,074 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:53,074 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:53:53,074 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:53:53,074 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:53,074 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,074 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:53,074 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:53,074 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:53,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:53,074 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:53,074 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:53,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:53,076 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:53:53,076 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:53:53,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:53,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:53,176 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:53:53,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:53:53,179 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:53:53,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:53:53,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:53:53,180 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:53:53,180 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:53,180 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:53:53,180 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:53:53,181 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:53,181 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:53,182 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,182 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:53,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:53,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:53,182 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,182 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:53:53,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:53,183 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:53,183 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:53,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54604 deadline: 1680281693181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:53:53,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:53,183 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:53:53,184 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:53,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54630 deadline: 1680281693182, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,184 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:53:53,184 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:53:53,184 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:53:53,184 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:53,184 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:53:53,184 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:53:53,184 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:53:53,184 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:53,184 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. started... 2023-03-31 16:53:53,185 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. as already flushing 2023-03-31 16:53:53,185 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:53,185 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:53:53,185 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:53:53,185 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,185 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:53:53,185 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:53:53,187 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,187 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:53,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:53,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:53,187 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:53,187 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:53:53,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:53,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:53,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:53,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:53,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:53:53,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a2514d2[Count = 0] remaining members to acquire global barrier 2023-03-31 16:53:53,189 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:53:53,189 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:53,190 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:53,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:53,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:53,191 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:53:53,191 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:53:53,191 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,191 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:53:53,191 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:53:53,192 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,192 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:53:53,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:53,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:53,192 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:53,193 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:53:53,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:53,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:53,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:53,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:53,194 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:53,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54628 deadline: 1680281693189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:53,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:53:53,195 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:53:53,195 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:53:53,195 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:53:53,195 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:53:53,195 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:53:53,195 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:53:53,197 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:53,197 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:53,197 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:53,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:53,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:53,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:53,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:53,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:53,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:53,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:53,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:53,197 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:53,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:53,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:53,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:53,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,199 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:53,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:53,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,204 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,204 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:53,204 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:53,205 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:53:53,205 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:53,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:53:53,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:53:53,205 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:53,205 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,205 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:53,205 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:53,205 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:53,205 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:53:53,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:53,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:53,205 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:53,205 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:53:53,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:53,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:53,298 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/A/dba0bdcef83a46b1abacb7a889a8f992 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/dba0bdcef83a46b1abacb7a889a8f992 2023-03-31 16:53:53,305 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:53:53,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:53:53,309 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:53:53,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:53:53,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:53:53,311 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:53:53,311 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:53,311 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:53:53,311 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:53:53,311 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in e94df50bb46b68756c6a704332e1c5f4/A of e94df50bb46b68756c6a704332e1c5f4 into dba0bdcef83a46b1abacb7a889a8f992(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:53:53,311 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e94df50bb46b68756c6a704332e1c5f4: 2023-03-31 16:53:53,312 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:53,312 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4., storeName=e94df50bb46b68756c6a704332e1c5f4/A, priority=11, startTime=1680281632784; duration=0sec 2023-03-31 16:53:53,312 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:53,314 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:53,314 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:53:53,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:53,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:53,314 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:53,315 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:53,315 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,315 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:53:53,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:53:53,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:53,316 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:53:53,316 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:53:53,316 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:53:53,316 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:53:53,317 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:53,317 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:53:53,318 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:53:53,318 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:53:53,318 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:53,318 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. started... 2023-03-31 16:53:53,318 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. as already flushing 2023-03-31 16:53:53,318 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:53,318 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:53:53,318 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:53:53,318 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,318 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:53:53,318 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:53:53,320 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:53,320 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:53,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:53,320 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:53,320 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:53:53,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:53,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:53,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:53,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:53,322 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:53:53,322 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e614521[Count = 0] remaining members to acquire global barrier 2023-03-31 16:53:53,322 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:53:53,322 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:53,323 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:53,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:53,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:53,324 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:53:53,324 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:53:53,324 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:53:53,325 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,325 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:53:53,330 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:53:53,330 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,330 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:53,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:53,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:53,330 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:53:53,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:53,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:53,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:53,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:53,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:53,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:53:53,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:53:53,333 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:53:53,333 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:53:53,333 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:53:53,333 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:53:53,333 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:53:53,336 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:53,336 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:53,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:53,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:53,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:53,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:53,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:53,338 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:53,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:53,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:53,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:53,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:53,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:53,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:53,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:53,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:53,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:53,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,342 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,346 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,349 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,349 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:53,349 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:53,349 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,349 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:53,349 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:53,349 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:53,349 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:53,349 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:53,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:53,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:53,350 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:53,350 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:53:53,350 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:53:53,350 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:53:53,350 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:53:53,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:53,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:53,371 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=282 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/C/84f9af705f0043659bad1f6f685eca61 2023-03-31 16:53:53,383 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/A/f802b521ea6349c78739bcfc3a5b5b21 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/f802b521ea6349c78739bcfc3a5b5b21 2023-03-31 16:53:53,392 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/f802b521ea6349c78739bcfc3a5b5b21, entries=250, sequenceid=282, filesize=16.7 K 2023-03-31 16:53:53,393 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/B/2c04ffc461254f4298677026794e0271 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/2c04ffc461254f4298677026794e0271 2023-03-31 16:53:53,400 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/2c04ffc461254f4298677026794e0271, entries=150, sequenceid=282, filesize=11.9 K 2023-03-31 16:53:53,401 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/C/84f9af705f0043659bad1f6f685eca61 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/84f9af705f0043659bad1f6f685eca61 2023-03-31 16:53:53,410 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/84f9af705f0043659bad1f6f685eca61, entries=150, sequenceid=282, filesize=11.9 K 2023-03-31 16:53:53,411 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~415.41 KB/425376, currentSize=40.25 KB/41220 for e94df50bb46b68756c6a704332e1c5f4 in 559ms, sequenceid=282, compaction requested=false 2023-03-31 16:53:53,411 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e94df50bb46b68756c6a704332e1c5f4: 2023-03-31 16:53:53,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on e94df50bb46b68756c6a704332e1c5f4 2023-03-31 16:53:53,421 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e94df50bb46b68756c6a704332e1c5f4 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-31 16:53:53,422 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=A 2023-03-31 16:53:53,422 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:53:53,422 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=B 2023-03-31 16:53:53,422 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:53:53,422 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=C 2023-03-31 16:53:53,422 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:53:53,450 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:53:53,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:53:53,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:53:53,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:53:53,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:53:53,460 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:53:53,460 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:53,460 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:53:53,460 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:53:53,461 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:53,461 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:53,463 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,463 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:53,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:53,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:53,463 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,463 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:53:53,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:53,464 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:53,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:53:53,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:53,465 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:53:53,465 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:53:53,465 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:53:53,465 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:53:53,465 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:53,466 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:53:53,467 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:53:53,467 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:53:53,467 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:53,468 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. started... 2023-03-31 16:53:53,468 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. as already flushing 2023-03-31 16:53:53,468 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:53,468 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:53:53,468 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:53:53,468 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,468 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:53:53,468 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:53:53,470 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,470 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:53,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:53,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:53,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:53,480 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:53,480 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:53:53,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:53,482 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=295 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/A/3c8e928d2d8e424489927535f482d620 2023-03-31 16:53:53,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:53,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:53,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:53:53,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d1b37d4[Count = 0] remaining members to acquire global barrier 2023-03-31 16:53:53,483 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:53:53,483 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:53,485 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:53,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:53,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:53,485 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:53:53,485 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:53:53,485 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:53:53,487 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,487 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:53:53,487 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:53,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:53,488 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:53:53,488 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:53,488 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:53:53,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:53,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:53,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:53,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:53,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:53,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:53:53,494 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:53:53,494 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:53:53,494 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:53:53,494 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:53:53,494 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:53:53,494 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:53:53,497 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:53,497 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:53,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:53,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:53,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:53,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:53,499 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:53,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:53,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:53,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:53,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:53,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:53,500 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:53,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:53,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:53,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,501 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:53,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:53,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,503 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:53,503 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:53,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:53,503 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:53,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:53:53,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:53:53,504 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:53,504 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:53,504 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:53,504 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,504 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:53,504 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:53:53,504 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:53,504 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:53,504 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:53:53,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:53,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:53,510 WARN [DataStreamer for file /user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/B/6212feb723dc4363a8b87771dd3f4107] hdfs.DataStreamer(982): Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1257) at java.lang.Thread.join(Thread.java:1331) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:980) at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:630) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:807) 2023-03-31 16:53:53,511 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680281633473, max timestamp=1680281633508Num uniques 859; 2023-03-31 16:53:53,511 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680281633473, max timestamp=1680281633508Num uniques 859; 2023-03-31 16:53:53,512 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-31 16:53:53,512 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-31 16:53:53,512 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680281633495, max timestamp=1680281633508Num uniques 859; 2023-03-31 16:53:53,513 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 16:53:53,514 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=295 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/B/6212feb723dc4363a8b87771dd3f4107 2023-03-31 16:53:53,532 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=295 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/C/52acedccffda4f928340331b4c057a30 2023-03-31 16:53:53,557 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:53,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54628 deadline: 1680281693531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,558 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:53,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54602 deadline: 1680281693533, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,559 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:53,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54604 deadline: 1680281693535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,561 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:53,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54630 deadline: 1680281693537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,561 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:53,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54638 deadline: 1680281693538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,563 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/A/3c8e928d2d8e424489927535f482d620 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/3c8e928d2d8e424489927535f482d620 2023-03-31 16:53:53,570 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/3c8e928d2d8e424489927535f482d620, entries=150, sequenceid=295, filesize=11.9 K 2023-03-31 16:53:53,570 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/B/6212feb723dc4363a8b87771dd3f4107 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/6212feb723dc4363a8b87771dd3f4107 2023-03-31 16:53:53,576 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/6212feb723dc4363a8b87771dd3f4107, entries=150, sequenceid=295, filesize=11.9 K 2023-03-31 16:53:53,577 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/C/52acedccffda4f928340331b4c057a30 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/52acedccffda4f928340331b4c057a30 2023-03-31 16:53:53,604 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/52acedccffda4f928340331b4c057a30, entries=150, sequenceid=295, filesize=11.9 K 2023-03-31 16:53:53,605 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:53:53,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:53:53,608 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:53:53,609 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=136.42 KB/139690 for e94df50bb46b68756c6a704332e1c5f4 in 188ms, sequenceid=295, compaction requested=true 2023-03-31 16:53:53,609 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e94df50bb46b68756c6a704332e1c5f4: 2023-03-31 16:53:53,610 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 16:53:53,610 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:53:53,610 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 16:53:53,611 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:53:53,611 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:53:53,630 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 41732 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 16:53:53,631 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1912): e94df50bb46b68756c6a704332e1c5f4/A is initiating minor compaction (all files) 2023-03-31 16:53:53,631 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e94df50bb46b68756c6a704332e1c5f4/A in TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:53,631 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/dba0bdcef83a46b1abacb7a889a8f992, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/f802b521ea6349c78739bcfc3a5b5b21, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/3c8e928d2d8e424489927535f482d620] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp, totalSize=40.8 K 2023-03-31 16:53:53,631 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36852 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 16:53:53,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:53:53,631 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): e94df50bb46b68756c6a704332e1c5f4/B is initiating minor compaction (all files) 2023-03-31 16:53:53,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:53:53,632 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e94df50bb46b68756c6a704332e1c5f4/B in TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:53,632 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:53:53,632 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:53,632 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/ad0cf48c84594732a4e59d5d88c010f4, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/2c04ffc461254f4298677026794e0271, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/6212feb723dc4363a8b87771dd3f4107] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp, totalSize=36.0 K 2023-03-31 16:53:53,632 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:53:53,632 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:53:53,632 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting dba0bdcef83a46b1abacb7a889a8f992, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=245, earliestPutTs=1680281631716 2023-03-31 16:53:53,634 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting ad0cf48c84594732a4e59d5d88c010f4, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=245, earliestPutTs=1680281631716 2023-03-31 16:53:53,634 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting f802b521ea6349c78739bcfc3a5b5b21, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=282, earliestPutTs=1680281632262 2023-03-31 16:53:53,634 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:53,634 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:53,634 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 2c04ffc461254f4298677026794e0271, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=282, earliestPutTs=1680281632262 2023-03-31 16:53:53,634 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 3c8e928d2d8e424489927535f482d620, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=295, earliestPutTs=1680281632863 2023-03-31 16:53:53,635 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 6212feb723dc4363a8b87771dd3f4107, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=295, earliestPutTs=1680281632863 2023-03-31 16:53:53,636 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:53,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:53,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:53,636 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:53,637 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,637 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:53:53,637 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:53,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:53:53,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:53,638 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:53:53,639 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:53:53,639 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:53:53,639 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:53,639 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:53:53,640 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:53:53,640 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:53:53,640 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:53,640 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. started... 2023-03-31 16:53:53,641 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e94df50bb46b68756c6a704332e1c5f4 3/3 column families, dataSize=136.42 KB heapSize=385.38 KB 2023-03-31 16:53:53,641 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=A 2023-03-31 16:53:53,642 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:53:53,642 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=B 2023-03-31 16:53:53,642 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:53:53,642 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=C 2023-03-31 16:53:53,642 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:53:53,647 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] throttle.PressureAwareThroughputController(145): e94df50bb46b68756c6a704332e1c5f4#A#compaction#48 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:53:53,651 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): e94df50bb46b68756c6a704332e1c5f4#B#compaction#49 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:53:53,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on e94df50bb46b68756c6a704332e1c5f4 2023-03-31 16:53:53,667 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. as already flushing 2023-03-31 16:53:53,676 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:53,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54638 deadline: 1680281693672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,678 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:53,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54630 deadline: 1680281693674, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,689 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:53,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54628 deadline: 1680281693675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,689 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:53,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54604 deadline: 1680281693677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,690 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:53,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54602 deadline: 1680281693678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,703 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=331 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/A/ecc3d3d5fe32409489544f737312592e 2023-03-31 16:53:53,750 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=331 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/B/3eb92bf6351448318f33cec6a39a01bb 2023-03-31 16:53:53,775 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=331 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/C/920249a17bfd4d89bfe479e022083798 2023-03-31 16:53:53,782 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/A/ecc3d3d5fe32409489544f737312592e as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/ecc3d3d5fe32409489544f737312592e 2023-03-31 16:53:53,786 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:53,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54638 deadline: 1680281693779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,786 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:53,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54630 deadline: 1680281693780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,788 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/ecc3d3d5fe32409489544f737312592e, entries=150, sequenceid=331, filesize=11.9 K 2023-03-31 16:53:53,791 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/B/3eb92bf6351448318f33cec6a39a01bb as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/3eb92bf6351448318f33cec6a39a01bb 2023-03-31 16:53:53,795 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:53,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54628 deadline: 1680281693790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,796 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:53,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54604 deadline: 1680281693791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,796 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:53,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54602 deadline: 1680281693791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,798 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/3eb92bf6351448318f33cec6a39a01bb, entries=150, sequenceid=331, filesize=11.9 K 2023-03-31 16:53:53,800 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/C/920249a17bfd4d89bfe479e022083798 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/920249a17bfd4d89bfe479e022083798 2023-03-31 16:53:53,805 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/920249a17bfd4d89bfe479e022083798, entries=150, sequenceid=331, filesize=11.9 K 2023-03-31 16:53:53,806 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~136.42 KB/139690, heapSize ~385.33 KB/394576, currentSize=53.67 KB/54960 for e94df50bb46b68756c6a704332e1c5f4 in 166ms, sequenceid=331, compaction requested=true 2023-03-31 16:53:53,807 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e94df50bb46b68756c6a704332e1c5f4: 2023-03-31 16:53:53,807 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:53,807 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:53:53,808 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:53:53,808 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,808 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:53:53,808 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:53:53,810 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,810 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:53,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:53,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:53,811 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:53,811 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:53:53,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:53,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:53,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:53,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:53,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:53:53,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2cc53303[Count = 0] remaining members to acquire global barrier 2023-03-31 16:53:53,813 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:53:53,813 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:53,814 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:53,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:53,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:53,815 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:53:53,815 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:53:53,815 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:53:53,815 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,815 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:53:53,816 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,816 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:53:53,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:53,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:53,816 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:53,816 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:53:53,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:53,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:53,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:53,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:53,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:53,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:53:53,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:53:53,818 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:53:53,819 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:53:53,819 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:53:53,819 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:53:53,819 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:53:53,820 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:53,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:53,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:53,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:53,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:53,821 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:53,821 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:53,821 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:53,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:53,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:53,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:53,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:53,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:53,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:53,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:53,823 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:53,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:53,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,826 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,826 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:53,827 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:53,827 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:53:53,827 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:53:53,827 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:53:53,827 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:53,827 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:53,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:53,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:53,827 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:53,827 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,827 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:53,827 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:53,827 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:53,827 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:53:53,827 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:53:53,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:53,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:53,928 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:53:53,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:53:53,930 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:53:53,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:53:53,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:53:53,932 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:53:53,933 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:53,933 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:53:53,933 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:53:53,933 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:53,933 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:53,935 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:53,935 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:53,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:53,935 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:53,935 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:53:53,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:53,935 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:53,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:53:53,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:53,936 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:53:53,936 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:53:53,936 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:53:53,936 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:53:53,936 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:53,936 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:53:53,937 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:53:53,937 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:53:53,937 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:53,937 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. started... 2023-03-31 16:53:53,937 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e94df50bb46b68756c6a704332e1c5f4 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-31 16:53:53,937 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=A 2023-03-31 16:53:53,937 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:53:53,937 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=B 2023-03-31 16:53:53,937 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:53:53,937 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=C 2023-03-31 16:53:53,937 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:53:53,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on e94df50bb46b68756c6a704332e1c5f4 2023-03-31 16:53:53,990 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. as already flushing 2023-03-31 16:53:54,018 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:54,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54630 deadline: 1680281694013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:54,018 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:54,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54604 deadline: 1680281694013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:54,019 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:54,019 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680281634002, max timestamp=1680281634016Num uniques 859; 2023-03-31 16:53:54,019 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680281634002, max timestamp=1680281634016Num uniques 859; 2023-03-31 16:53:54,019 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680281634002, max timestamp=1680281634016Num uniques 859; 2023-03-31 16:53:54,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54638 deadline: 1680281694014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:54,019 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-31 16:53:54,019 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-31 16:53:54,019 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-31 16:53:54,058 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:54,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54602 deadline: 1680281694053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:54,059 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:54,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54628 deadline: 1680281694054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:54,077 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/A/1645bd9057cf4a9c922573079eca9086 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/1645bd9057cf4a9c922573079eca9086 2023-03-31 16:53:54,084 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e94df50bb46b68756c6a704332e1c5f4/A of e94df50bb46b68756c6a704332e1c5f4 into 1645bd9057cf4a9c922573079eca9086(size=12.4 K), total size for store is 24.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:53:54,084 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e94df50bb46b68756c6a704332e1c5f4: 2023-03-31 16:53:54,084 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4., storeName=e94df50bb46b68756c6a704332e1c5f4/A, priority=13, startTime=1680281633609; duration=0sec 2023-03-31 16:53:54,084 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:53:54,084 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 16:53:54,086 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49038 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 16:53:54,086 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1912): e94df50bb46b68756c6a704332e1c5f4/C is initiating minor compaction (all files) 2023-03-31 16:53:54,086 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e94df50bb46b68756c6a704332e1c5f4/C in TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:54,086 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/c87ddfb34b7f4b49a8a29c3211d6972a, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/84f9af705f0043659bad1f6f685eca61, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/52acedccffda4f928340331b4c057a30, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/920249a17bfd4d89bfe479e022083798] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp, totalSize=47.9 K 2023-03-31 16:53:54,087 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting c87ddfb34b7f4b49a8a29c3211d6972a, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=245, earliestPutTs=1680281631716 2023-03-31 16:53:54,087 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 84f9af705f0043659bad1f6f685eca61, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=282, earliestPutTs=1680281632262 2023-03-31 16:53:54,088 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 52acedccffda4f928340331b4c057a30, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=295, earliestPutTs=1680281632863 2023-03-31 16:53:54,089 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 920249a17bfd4d89bfe479e022083798, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=331, earliestPutTs=1680281633521 2023-03-31 16:53:54,099 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/B/1864df0ce2894eb288cbfff2374a9af3 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/1864df0ce2894eb288cbfff2374a9af3 2023-03-31 16:53:54,111 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e94df50bb46b68756c6a704332e1c5f4/B of e94df50bb46b68756c6a704332e1c5f4 into 1864df0ce2894eb288cbfff2374a9af3(size=12.4 K), total size for store is 24.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:53:54,111 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e94df50bb46b68756c6a704332e1c5f4: 2023-03-31 16:53:54,111 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] throttle.PressureAwareThroughputController(145): e94df50bb46b68756c6a704332e1c5f4#C#compaction#54 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:53:54,111 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4., storeName=e94df50bb46b68756c6a704332e1c5f4/B, priority=13, startTime=1680281633610; duration=0sec 2023-03-31 16:53:54,111 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:53:54,124 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/C/62110af81f184712ad28ed979415eb2b as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/62110af81f184712ad28ed979415eb2b 2023-03-31 16:53:54,124 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:54,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54604 deadline: 1680281694121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:54,125 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:54,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54630 deadline: 1680281694121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:54,125 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:54,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54638 deadline: 1680281694121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:54,131 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e94df50bb46b68756c6a704332e1c5f4/C of e94df50bb46b68756c6a704332e1c5f4 into 62110af81f184712ad28ed979415eb2b(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:53:54,131 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e94df50bb46b68756c6a704332e1c5f4: 2023-03-31 16:53:54,131 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4., storeName=e94df50bb46b68756c6a704332e1c5f4/C, priority=12, startTime=1680281633611; duration=0sec 2023-03-31 16:53:54,132 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:53:54,165 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:54,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54602 deadline: 1680281694160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:54,165 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:54,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54628 deadline: 1680281694160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:54,328 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:54,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54604 deadline: 1680281694327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:54,328 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:54,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54630 deadline: 1680281694328, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:54,329 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:54,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54638 deadline: 1680281694328, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:54,359 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=342 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/A/f5f76c86ce5f49d98e21b7df754e2b9c 2023-03-31 16:53:54,371 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:54,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54628 deadline: 1680281694367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:54,373 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:54,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54602 deadline: 1680281694369, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:54,631 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:54,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54604 deadline: 1680281694630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:54,631 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:54,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54638 deadline: 1680281694631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:54,632 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:54,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54630 deadline: 1680281694631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:54,676 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:54,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54628 deadline: 1680281694675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:54,677 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:54,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54602 deadline: 1680281694677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:54,775 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=342 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/B/b11fba94fc2248498af9922a921794cf 2023-03-31 16:53:55,137 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:55,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54638 deadline: 1680281695135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:55,138 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:55,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54604 deadline: 1680281695136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:55,139 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:55,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54630 deadline: 1680281695136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:55,180 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:55,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54628 deadline: 1680281695180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:55,185 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:55,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54602 deadline: 1680281695180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:55,195 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=342 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/C/1011fd0251c1431d92200dd2da9e5622 2023-03-31 16:53:55,203 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/A/f5f76c86ce5f49d98e21b7df754e2b9c as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/f5f76c86ce5f49d98e21b7df754e2b9c 2023-03-31 16:53:55,212 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/f5f76c86ce5f49d98e21b7df754e2b9c, entries=150, sequenceid=342, filesize=11.9 K 2023-03-31 16:53:55,213 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/B/b11fba94fc2248498af9922a921794cf as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/b11fba94fc2248498af9922a921794cf 2023-03-31 16:53:55,225 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/b11fba94fc2248498af9922a921794cf, entries=150, sequenceid=342, filesize=11.9 K 2023-03-31 16:53:55,228 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/C/1011fd0251c1431d92200dd2da9e5622 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/1011fd0251c1431d92200dd2da9e5622 2023-03-31 16:53:55,239 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/1011fd0251c1431d92200dd2da9e5622, entries=150, sequenceid=342, filesize=11.9 K 2023-03-31 16:53:55,240 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for e94df50bb46b68756c6a704332e1c5f4 in 1303ms, sequenceid=342, compaction requested=true 2023-03-31 16:53:55,240 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e94df50bb46b68756c6a704332e1c5f4: 2023-03-31 16:53:55,240 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:55,240 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:53:55,240 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:53:55,240 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:55,240 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:53:55,240 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:53:55,245 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:55,245 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:55,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:55,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:55,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:55,245 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:55,245 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:53:55,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:55,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:55,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:55,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:55,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:55,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:53:55,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e7431cd[Count = 0] remaining members to acquire global barrier 2023-03-31 16:53:55,247 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:53:55,247 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:55,248 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:55,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:55,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:55,249 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:53:55,249 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:53:55,249 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:53:55,249 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:55,249 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:53:55,250 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:55,250 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:53:55,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:55,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:55,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:55,250 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:55,251 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:53:55,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:55,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:55,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:55,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:55,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:55,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:55,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:55,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:53:55,257 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:53:55,258 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:53:55,258 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:53:55,258 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:53:55,258 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:53:55,258 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:53:55,260 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:55,260 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:55,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:55,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:55,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:55,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:55,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:55,262 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:55,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:55,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:55,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:55,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:55,263 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:55,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:55,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:55,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:55,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:55,265 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:55,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:55,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:55,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:55,269 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:55,269 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:55,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:55,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:55,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:55,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:55,270 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:55,271 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:55,271 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:55,271 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:53:55,271 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:53:55,271 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:55,271 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:55,271 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:55,271 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:55,271 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:55,271 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:53:55,271 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:53:55,371 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:53:55,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:53:55,379 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:53:55,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:53:55,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:53:55,380 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:53:55,380 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:55,380 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:53:55,380 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:53:55,380 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:55,381 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:55,382 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:55,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:55,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:55,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:55,383 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:55,383 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:55,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:53:55,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:55,383 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:53:55,384 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:55,384 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:53:55,392 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:53:55,392 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:53:55,393 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:53:55,393 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:55,393 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:53:55,397 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:53:55,397 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:53:55,397 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:55,397 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. started... 2023-03-31 16:53:55,397 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e94df50bb46b68756c6a704332e1c5f4 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-03-31 16:53:55,398 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=A 2023-03-31 16:53:55,398 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:53:55,398 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=B 2023-03-31 16:53:55,398 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:53:55,398 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=C 2023-03-31 16:53:55,398 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:53:55,827 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=379 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/A/c6973cb572fa4b519ca3aaad2a78c689 2023-03-31 16:53:55,845 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=379 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/B/ed0f0ed9facd4225b2365b8cb81f88b7 2023-03-31 16:53:55,863 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=379 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/C/d2d0a538cfd74c61bdd9d06edf1d1fc9 2023-03-31 16:53:55,870 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/A/c6973cb572fa4b519ca3aaad2a78c689 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/c6973cb572fa4b519ca3aaad2a78c689 2023-03-31 16:53:55,877 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/c6973cb572fa4b519ca3aaad2a78c689, entries=150, sequenceid=379, filesize=11.9 K 2023-03-31 16:53:55,880 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/B/ed0f0ed9facd4225b2365b8cb81f88b7 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/ed0f0ed9facd4225b2365b8cb81f88b7 2023-03-31 16:53:55,893 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/ed0f0ed9facd4225b2365b8cb81f88b7, entries=150, sequenceid=379, filesize=11.9 K 2023-03-31 16:53:55,894 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/C/d2d0a538cfd74c61bdd9d06edf1d1fc9 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/d2d0a538cfd74c61bdd9d06edf1d1fc9 2023-03-31 16:53:55,901 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/d2d0a538cfd74c61bdd9d06edf1d1fc9, entries=150, sequenceid=379, filesize=11.9 K 2023-03-31 16:53:55,903 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=0 B/0 for e94df50bb46b68756c6a704332e1c5f4 in 506ms, sequenceid=379, compaction requested=true 2023-03-31 16:53:55,903 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e94df50bb46b68756c6a704332e1c5f4: 2023-03-31 16:53:55,903 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:55,903 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:53:55,903 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:53:55,903 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:55,903 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:53:55,905 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:53:55,911 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:55,911 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:55,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:55,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:55,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:55,912 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:55,912 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:53:55,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:55,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:55,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:55,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:55,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:55,913 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:53:55,913 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ab8376f[Count = 0] remaining members to acquire global barrier 2023-03-31 16:53:55,913 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:53:55,914 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:55,915 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:55,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:55,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:55,915 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:53:55,915 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:53:55,915 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:55,915 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:53:55,915 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:53:55,917 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:53:55,917 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:55,917 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:55,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:55,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:55,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:55,917 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:53:55,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:55,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:55,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:55,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:55,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:55,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:55,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:55,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:53:55,919 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:53:55,919 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:53:55,920 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:53:55,920 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:53:55,920 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:53:55,920 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:53:55,921 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:55,921 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:55,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:55,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:55,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:55,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:55,921 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:55,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:55,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:55,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:55,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:55,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:55,924 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:55,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:55,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:55,925 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:55,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:55,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:55,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:55,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:55,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:55,929 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:55,929 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:55,929 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:55,929 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:55,929 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:55,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:55,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:55,929 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:53:55,929 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:53:55,929 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:55,929 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:55,930 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:55,930 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:55,930 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:55,930 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:53:55,930 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:53:55,930 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:53:55,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:55,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:56,030 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:53:56,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:53:56,033 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:53:56,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:53:56,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:53:56,034 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:53:56,034 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:56,034 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:53:56,034 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:53:56,035 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:56,035 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:56,036 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:56,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:56,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:56,036 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:56,037 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:56,038 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,038 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:53:56,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:53:56,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:56,038 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:53:56,039 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:53:56,039 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:53:56,039 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:53:56,039 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:56,039 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:53:56,041 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:53:56,041 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:53:56,045 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:56,045 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. started... 2023-03-31 16:53:56,045 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e94df50bb46b68756c6a704332e1c5f4: 2023-03-31 16:53:56,045 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:56,045 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:53:56,045 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:53:56,045 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,046 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:53:56,046 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:53:56,047 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,047 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:56,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:56,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:56,048 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:56,048 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:53:56,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:56,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:56,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:56,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:56,050 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:53:56,050 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e69167b[Count = 0] remaining members to acquire global barrier 2023-03-31 16:53:56,050 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:53:56,050 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:56,052 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:56,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:56,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:56,052 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:53:56,052 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:53:56,052 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:53:56,054 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,054 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:53:56,054 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:56,054 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:53:56,054 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:53:56,055 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:53:56,055 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:53:56,055 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:53:56,055 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:53:56,055 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:53:56,055 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:53:56,057 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:56,057 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:56,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:56,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:56,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:56,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:56,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:56,057 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:56,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:56,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:56,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:56,058 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:56,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:56,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:56,060 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:56,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:56,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:56,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,062 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,062 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:56,062 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:56,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:53:56,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:53:56,062 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:56,063 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,063 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:56,063 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:53:56,063 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:53:56,063 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:56,063 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:56,063 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:56,063 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:56,063 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:53:56,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:56,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:56,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:56,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:56,163 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:53:56,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:53:56,168 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:53:56,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on e94df50bb46b68756c6a704332e1c5f4 2023-03-31 16:53:56,174 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e94df50bb46b68756c6a704332e1c5f4 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-31 16:53:56,175 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=A 2023-03-31 16:53:56,175 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:53:56,176 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=B 2023-03-31 16:53:56,176 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:53:56,176 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=C 2023-03-31 16:53:56,176 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:53:56,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:53:56,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:53:56,180 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:53:56,180 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:56,180 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:53:56,180 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:53:56,181 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:56,181 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:56,182 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,182 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,182 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:53:56,184 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:56,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:56,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:56,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:56,186 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:56,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:53:56,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:56,186 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:53:56,188 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:53:56,188 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:53:56,188 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:53:56,188 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:56,188 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:53:56,195 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:53:56,195 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:53:56,195 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:56,195 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. started... 2023-03-31 16:53:56,196 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. as already flushing 2023-03-31 16:53:56,196 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:56,196 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:53:56,196 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:53:56,196 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,197 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:53:56,197 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:53:56,199 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,199 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:56,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:56,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:56,200 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:56,200 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:53:56,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:56,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:56,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:56,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:56,202 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:53:56,202 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b317d9e[Count = 0] remaining members to acquire global barrier 2023-03-31 16:53:56,202 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:53:56,202 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:56,206 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:56,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:56,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:56,206 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:53:56,206 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:53:56,206 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:53:56,206 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,206 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:53:56,208 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,208 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:53:56,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:56,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:56,208 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:56,208 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:53:56,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:56,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:56,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:56,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:56,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:56,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:53:56,211 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:53:56,211 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:53:56,211 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:53:56,211 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:53:56,211 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:53:56,211 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:53:56,217 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:56,217 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:56,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:56,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:56,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:56,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:56,217 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:56,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:56,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:56,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:56,218 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:56,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:56,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:56,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:56,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:56,220 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:56,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:56,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,224 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:56,224 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:56,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:56,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:56,224 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,224 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:56,224 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:56,225 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,225 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:56,225 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:53:56,225 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:53:56,225 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:56,225 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:56,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:56,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:56,225 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:53:56,225 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:53:56,225 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:56,241 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680281636227, max timestamp=1680281636239Num uniques 859; 2023-03-31 16:53:56,241 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 16:53:56,241 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680281636227, max timestamp=1680281636239Num uniques 859; 2023-03-31 16:53:56,241 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680281636227, max timestamp=1680281636239Num uniques 859; 2023-03-31 16:53:56,241 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 16:53:56,241 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 16:53:56,265 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:56,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54604 deadline: 1680281696257, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,265 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:56,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54628 deadline: 1680281696257, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,266 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:56,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54630 deadline: 1680281696258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,269 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:56,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54602 deadline: 1680281696263, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,269 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:56,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54638 deadline: 1680281696264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,325 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:53:56,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:53:56,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:53:56,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:53:56,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:53:56,329 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:53:56,329 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:56,329 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:53:56,329 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:53:56,330 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:56,330 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:56,331 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,331 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:56,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:56,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:56,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:56,332 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:56,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:53:56,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:56,333 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:53:56,333 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:53:56,333 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:53:56,333 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:53:56,333 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:56,333 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:53:56,333 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:53:56,333 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:53:56,333 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:56,333 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. started... 2023-03-31 16:53:56,333 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. as already flushing 2023-03-31 16:53:56,334 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:56,334 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:53:56,334 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:53:56,334 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,334 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:53:56,334 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:53:56,336 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,336 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:53:56,337 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,337 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:56,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:56,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:56,337 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:56,337 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:53:56,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:56,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:56,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:56,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:56,339 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:53:56,339 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5474c36c[Count = 0] remaining members to acquire global barrier 2023-03-31 16:53:56,339 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:53:56,339 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:56,340 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:56,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:56,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:56,340 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:53:56,340 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:53:56,340 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:53:56,340 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,341 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:53:56,343 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,343 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:53:56,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:56,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:56,343 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:56,343 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:53:56,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:56,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:56,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:56,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:56,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:56,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:53:56,346 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:53:56,346 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:53:56,346 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:53:56,346 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:53:56,346 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:53:56,346 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:53:56,348 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:56,348 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:56,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:56,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:56,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:56,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:56,348 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:56,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:56,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:56,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:56,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:56,351 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:56,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:56,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:56,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:56,353 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:56,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:56,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,355 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:56,356 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:56,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:56,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:56,356 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,356 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:56,356 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:56,356 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:56,356 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:56,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:56,356 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:56,356 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:53:56,356 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:56,356 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:53:56,357 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:56,357 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:53:56,357 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:53:56,357 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:53:56,369 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:56,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54604 deadline: 1680281696367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,369 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:56,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54628 deadline: 1680281696367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,369 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:56,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54630 deadline: 1680281696367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,373 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:56,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54638 deadline: 1680281696370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,374 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:56,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54602 deadline: 1680281696371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,457 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:53:56,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:53:56,460 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:53:56,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:53:56,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:53:56,461 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:53:56,461 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:56,461 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:53:56,461 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:53:56,462 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:56,462 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:56,463 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:56,463 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:56,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:56,463 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,463 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:53:56,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:56,464 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:56,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:53:56,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:56,464 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:53:56,464 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:53:56,465 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:53:56,465 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:53:56,465 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:56,465 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:53:56,465 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:53:56,465 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:56,465 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. started... 2023-03-31 16:53:56,465 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:53:56,465 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. as already flushing 2023-03-31 16:53:56,465 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:56,465 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:53:56,465 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:53:56,465 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,465 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:53:56,465 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:53:56,468 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,468 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:56,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:56,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:56,469 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:56,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:56,469 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:53:56,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:56,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:56,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:56,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:53:56,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f2c85c2[Count = 0] remaining members to acquire global barrier 2023-03-31 16:53:56,470 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:53:56,470 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:56,471 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:56,472 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,472 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:53:56,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:56,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:56,476 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:53:56,476 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:53:56,476 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:53:56,478 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,478 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:53:56,478 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:56,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:56,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:56,478 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:53:56,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:56,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:56,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:56,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:56,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:56,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:53:56,481 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:53:56,481 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:53:56,481 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:53:56,481 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:53:56,481 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:53:56,481 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:53:56,483 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:56,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:56,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:56,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:56,483 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:56,483 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:56,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:56,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:56,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:56,488 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:56,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:56,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:56,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:56,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:56,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:56,489 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:56,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:56,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,492 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,492 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:56,492 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:56,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:56,492 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:56,493 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,492 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:56,493 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:56,493 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:56,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:56,493 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:56,493 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:56,493 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:53:56,493 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:53:56,493 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:53:56,493 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:53:56,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:56,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:56,493 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:53:56,573 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:56,573 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:56,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54628 deadline: 1680281696570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54630 deadline: 1680281696570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,574 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:56,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54604 deadline: 1680281696571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,579 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:56,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54638 deadline: 1680281696575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,585 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:56,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54602 deadline: 1680281696580, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,593 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:53:56,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:53:56,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:53:56,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:53:56,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:53:56,598 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:53:56,598 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:56,598 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:53:56,598 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:53:56,599 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:56,599 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:56,601 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=391 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/A/7f01d1fed2d54215b165088dde7c3a49 2023-03-31 16:53:56,601 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:56,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:56,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:56,602 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:56,602 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:56,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:53:56,603 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,603 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:53:56,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:56,604 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:53:56,604 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:53:56,604 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:53:56,604 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:53:56,604 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:56,604 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:53:56,608 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:53:56,608 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:53:56,608 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:56,608 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. started... 2023-03-31 16:53:56,608 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. as already flushing 2023-03-31 16:53:56,608 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:56,608 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:53:56,608 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:53:56,608 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,608 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:53:56,608 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:53:56,610 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,610 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:56,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:56,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:56,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:56,611 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:56,611 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:53:56,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:56,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:56,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:56,612 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:53:56,612 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29509092[Count = 0] remaining members to acquire global barrier 2023-03-31 16:53:56,612 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:53:56,613 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:56,614 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:56,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:56,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:56,614 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:53:56,614 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:53:56,614 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:53:56,614 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,614 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:53:56,619 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,619 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:53:56,620 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:56,620 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:53:56,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:56,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:56,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:56,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:56,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:56,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:56,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:56,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:53:56,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:53:56,623 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:53:56,623 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:53:56,623 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:53:56,623 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:53:56,623 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:53:56,627 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:56,627 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:56,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:56,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:56,627 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:56,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:56,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:56,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:56,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:56,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:56,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:56,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:56,628 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:56,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:56,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:56,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,630 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:56,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:56,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,634 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,634 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:56,634 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:56,634 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,634 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:56,634 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:56,634 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:56,635 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:56,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:53:56,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:53:56,635 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:56,635 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:56,637 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:53:56,637 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:53:56,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:56,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:56,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:56,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:56,647 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=391 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/B/3f8bb70b34ba4abbb3f7421ba3682087 2023-03-31 16:53:56,675 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=391 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/C/c9665615bee945e8b1fe6e6e2564c3c1 2023-03-31 16:53:56,682 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/A/7f01d1fed2d54215b165088dde7c3a49 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/7f01d1fed2d54215b165088dde7c3a49 2023-03-31 16:53:56,688 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/7f01d1fed2d54215b165088dde7c3a49, entries=250, sequenceid=391, filesize=16.7 K 2023-03-31 16:53:56,689 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/B/3f8bb70b34ba4abbb3f7421ba3682087 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/3f8bb70b34ba4abbb3f7421ba3682087 2023-03-31 16:53:56,694 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/3f8bb70b34ba4abbb3f7421ba3682087, entries=150, sequenceid=391, filesize=11.9 K 2023-03-31 16:53:56,695 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/C/c9665615bee945e8b1fe6e6e2564c3c1 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/c9665615bee945e8b1fe6e6e2564c3c1 2023-03-31 16:53:56,703 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/c9665615bee945e8b1fe6e6e2564c3c1, entries=150, sequenceid=391, filesize=11.9 K 2023-03-31 16:53:56,704 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for e94df50bb46b68756c6a704332e1c5f4 in 530ms, sequenceid=391, compaction requested=true 2023-03-31 16:53:56,704 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e94df50bb46b68756c6a704332e1c5f4: 2023-03-31 16:53:56,704 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:53:56,704 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-31 16:53:56,704 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-03-31 16:53:56,705 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-31 16:53:56,705 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-31 16:53:56,708 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61476 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-31 16:53:56,708 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): e94df50bb46b68756c6a704332e1c5f4/B is initiating minor compaction (all files) 2023-03-31 16:53:56,708 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e94df50bb46b68756c6a704332e1c5f4/B in TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:56,708 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/1864df0ce2894eb288cbfff2374a9af3, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/3eb92bf6351448318f33cec6a39a01bb, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/b11fba94fc2248498af9922a921794cf, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/ed0f0ed9facd4225b2365b8cb81f88b7, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/3f8bb70b34ba4abbb3f7421ba3682087] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp, totalSize=60.0 K 2023-03-31 16:53:56,709 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 1864df0ce2894eb288cbfff2374a9af3, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=295, earliestPutTs=1680281632863 2023-03-31 16:53:56,709 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 3eb92bf6351448318f33cec6a39a01bb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=331, earliestPutTs=1680281633521 2023-03-31 16:53:56,710 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting b11fba94fc2248498af9922a921794cf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=342, earliestPutTs=1680281633672 2023-03-31 16:53:56,711 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting ed0f0ed9facd4225b2365b8cb81f88b7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=379, earliestPutTs=1680281634028 2023-03-31 16:53:56,712 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 3f8bb70b34ba4abbb3f7421ba3682087, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=391, earliestPutTs=1680281636147 2023-03-31 16:53:56,717 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 66356 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-31 16:53:56,717 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1912): e94df50bb46b68756c6a704332e1c5f4/A is initiating minor compaction (all files) 2023-03-31 16:53:56,717 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e94df50bb46b68756c6a704332e1c5f4/A in TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:56,717 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/1645bd9057cf4a9c922573079eca9086, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/ecc3d3d5fe32409489544f737312592e, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/f5f76c86ce5f49d98e21b7df754e2b9c, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/c6973cb572fa4b519ca3aaad2a78c689, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/7f01d1fed2d54215b165088dde7c3a49] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp, totalSize=64.8 K 2023-03-31 16:53:56,718 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 1645bd9057cf4a9c922573079eca9086, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=295, earliestPutTs=1680281632863 2023-03-31 16:53:56,719 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting ecc3d3d5fe32409489544f737312592e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=331, earliestPutTs=1680281633521 2023-03-31 16:53:56,720 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting f5f76c86ce5f49d98e21b7df754e2b9c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=342, earliestPutTs=1680281633672 2023-03-31 16:53:56,725 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting c6973cb572fa4b519ca3aaad2a78c689, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=379, earliestPutTs=1680281634028 2023-03-31 16:53:56,726 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 7f01d1fed2d54215b165088dde7c3a49, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=391, earliestPutTs=1680281636147 2023-03-31 16:53:56,737 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:53:56,739 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): e94df50bb46b68756c6a704332e1c5f4#B#compaction#63 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:53:56,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:53:56,743 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:53:56,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:53:56,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:53:56,757 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:53:56,757 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:56,757 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:53:56,757 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:53:56,758 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:56,758 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:56,759 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:56,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:56,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:56,759 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:56,760 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,760 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:53:56,760 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:56,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:53:56,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:56,761 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:53:56,771 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:53:56,771 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:53:56,771 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:53:56,772 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:56,772 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:53:56,774 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] throttle.PressureAwareThroughputController(145): e94df50bb46b68756c6a704332e1c5f4#A#compaction#64 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:53:56,776 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:53:56,776 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:56,776 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:53:56,776 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. started... 2023-03-31 16:53:56,776 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e94df50bb46b68756c6a704332e1c5f4 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-03-31 16:53:56,776 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=A 2023-03-31 16:53:56,777 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:53:56,777 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=B 2023-03-31 16:53:56,777 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:53:56,777 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=C 2023-03-31 16:53:56,777 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:53:56,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on e94df50bb46b68756c6a704332e1c5f4 2023-03-31 16:53:56,879 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. as already flushing 2023-03-31 16:53:56,919 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:56,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54630 deadline: 1680281696910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,920 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:56,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54638 deadline: 1680281696912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,923 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:56,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54602 deadline: 1680281696914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,924 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:56,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54628 deadline: 1680281696918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:56,924 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:56,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54604 deadline: 1680281696919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:57,022 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:57,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54630 deadline: 1680281697021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:57,023 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:57,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54638 deadline: 1680281697021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:57,027 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:57,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54602 deadline: 1680281697025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:57,030 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:57,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54604 deadline: 1680281697027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:57,030 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:57,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54628 deadline: 1680281697028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:57,194 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/B/22eeaae46aaa4aedac30fc05e2a1fcdd as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/22eeaae46aaa4aedac30fc05e2a1fcdd 2023-03-31 16:53:57,203 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in e94df50bb46b68756c6a704332e1c5f4/B of e94df50bb46b68756c6a704332e1c5f4 into 22eeaae46aaa4aedac30fc05e2a1fcdd(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:53:57,203 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e94df50bb46b68756c6a704332e1c5f4: 2023-03-31 16:53:57,203 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4., storeName=e94df50bb46b68756c6a704332e1c5f4/B, priority=11, startTime=1680281636704; duration=0sec 2023-03-31 16:53:57,203 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:53:57,203 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 16:53:57,204 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49324 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 16:53:57,205 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): e94df50bb46b68756c6a704332e1c5f4/C is initiating minor compaction (all files) 2023-03-31 16:53:57,205 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e94df50bb46b68756c6a704332e1c5f4/C in TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:57,205 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/62110af81f184712ad28ed979415eb2b, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/1011fd0251c1431d92200dd2da9e5622, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/d2d0a538cfd74c61bdd9d06edf1d1fc9, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/c9665615bee945e8b1fe6e6e2564c3c1] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp, totalSize=48.2 K 2023-03-31 16:53:57,205 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 62110af81f184712ad28ed979415eb2b, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=331, earliestPutTs=1680281633521 2023-03-31 16:53:57,206 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 1011fd0251c1431d92200dd2da9e5622, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=342, earliestPutTs=1680281633672 2023-03-31 16:53:57,206 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting d2d0a538cfd74c61bdd9d06edf1d1fc9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=379, earliestPutTs=1680281634028 2023-03-31 16:53:57,207 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting c9665615bee945e8b1fe6e6e2564c3c1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=391, earliestPutTs=1680281636147 2023-03-31 16:53:57,215 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=427 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/A/221e1ab33201446c9815b31c933a0f71 2023-03-31 16:53:57,217 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): e94df50bb46b68756c6a704332e1c5f4#C#compaction#66 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:53:57,218 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/A/940492fd0c174e0286672200d5b9b2cd as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/940492fd0c174e0286672200d5b9b2cd 2023-03-31 16:53:57,225 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:57,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54638 deadline: 1680281697225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:57,227 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:57,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54630 deadline: 1680281697225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:57,230 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in e94df50bb46b68756c6a704332e1c5f4/A of e94df50bb46b68756c6a704332e1c5f4 into 940492fd0c174e0286672200d5b9b2cd(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:53:57,230 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e94df50bb46b68756c6a704332e1c5f4: 2023-03-31 16:53:57,230 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4., storeName=e94df50bb46b68756c6a704332e1c5f4/A, priority=11, startTime=1680281636704; duration=0sec 2023-03-31 16:53:57,230 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:53:57,231 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:57,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54602 deadline: 1680281697229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:57,234 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:57,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54604 deadline: 1680281697231, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:57,237 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:57,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54628 deadline: 1680281697232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:57,241 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=427 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/B/d6a82a8c0157451eb9050e6a57525e2b 2023-03-31 16:53:57,530 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:57,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54638 deadline: 1680281697528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:57,536 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:57,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54602 deadline: 1680281697533, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:57,536 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:57,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54630 deadline: 1680281697535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:57,539 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:57,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54604 deadline: 1680281697537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:57,539 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:57,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54628 deadline: 1680281697538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:57,633 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/C/1c85f91970d54826961dc4d064feb5a4 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/1c85f91970d54826961dc4d064feb5a4 2023-03-31 16:53:57,642 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e94df50bb46b68756c6a704332e1c5f4/C of e94df50bb46b68756c6a704332e1c5f4 into 1c85f91970d54826961dc4d064feb5a4(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:53:57,642 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e94df50bb46b68756c6a704332e1c5f4: 2023-03-31 16:53:57,642 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4., storeName=e94df50bb46b68756c6a704332e1c5f4/C, priority=12, startTime=1680281636704; duration=0sec 2023-03-31 16:53:57,642 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:53:57,655 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=427 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/C/01bb121ed8c048b99bb20915f4faf16d 2023-03-31 16:53:57,660 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/A/221e1ab33201446c9815b31c933a0f71 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/221e1ab33201446c9815b31c933a0f71 2023-03-31 16:53:57,665 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/221e1ab33201446c9815b31c933a0f71, entries=150, sequenceid=427, filesize=11.9 K 2023-03-31 16:53:57,666 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/B/d6a82a8c0157451eb9050e6a57525e2b as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/d6a82a8c0157451eb9050e6a57525e2b 2023-03-31 16:53:57,670 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/d6a82a8c0157451eb9050e6a57525e2b, entries=150, sequenceid=427, filesize=11.9 K 2023-03-31 16:53:57,672 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/C/01bb121ed8c048b99bb20915f4faf16d as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/01bb121ed8c048b99bb20915f4faf16d 2023-03-31 16:53:57,677 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/01bb121ed8c048b99bb20915f4faf16d, entries=150, sequenceid=427, filesize=11.9 K 2023-03-31 16:53:57,678 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for e94df50bb46b68756c6a704332e1c5f4 in 902ms, sequenceid=427, compaction requested=false 2023-03-31 16:53:57,678 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e94df50bb46b68756c6a704332e1c5f4: 2023-03-31 16:53:57,678 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:57,678 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:53:57,678 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:53:57,678 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:57,678 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:53:57,678 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:53:57,680 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:57,680 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:57,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:57,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:57,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:57,681 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:57,681 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:53:57,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:57,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:57,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:57,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:57,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:57,682 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:53:57,682 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49edd5ab[Count = 0] remaining members to acquire global barrier 2023-03-31 16:53:57,682 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:53:57,682 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:57,683 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:57,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:57,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:57,684 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:53:57,684 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:53:57,684 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:53:57,684 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:57,684 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:53:57,685 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:57,685 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:53:57,685 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:57,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:57,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:57,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:57,685 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:53:57,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:57,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:57,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:57,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:57,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:57,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:57,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:57,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:53:57,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:53:57,688 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:53:57,688 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:53:57,688 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:53:57,688 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:53:57,688 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:53:57,690 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:57,690 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:57,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:57,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:57,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:57,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:57,690 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:57,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:57,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:57,691 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:57,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:57,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:57,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:57,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:57,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:57,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:57,692 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:57,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:57,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:57,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:57,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:57,696 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:57,696 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:57,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:57,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:53:57,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:53:57,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:57,696 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:57,696 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:57,696 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:57,696 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:57,696 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:53:57,696 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:53:57,696 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:57,697 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:53:57,697 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:57,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:57,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:57,697 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:57,697 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:57,797 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:53:57,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:53:57,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:53:57,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:53:57,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:53:57,801 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:53:57,801 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:57,801 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:53:57,801 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:53:57,802 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:57,802 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:57,803 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:57,803 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:57,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:57,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:57,803 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:57,804 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:53:57,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:57,804 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:57,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:53:57,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:57,805 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:53:57,805 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:53:57,805 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:53:57,805 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:53:57,805 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:57,805 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:53:57,805 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:53:57,806 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:53:57,806 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:57,806 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. started... 2023-03-31 16:53:57,806 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e94df50bb46b68756c6a704332e1c5f4 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-31 16:53:57,806 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=A 2023-03-31 16:53:57,806 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:53:57,806 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=B 2023-03-31 16:53:57,806 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:53:57,806 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=C 2023-03-31 16:53:57,807 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:53:57,825 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=441 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/A/fe6024bfe55a48a29d3eb5e32c7899d8 2023-03-31 16:53:58,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on e94df50bb46b68756c6a704332e1c5f4 2023-03-31 16:53:58,035 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. as already flushing 2023-03-31 16:53:58,059 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:58,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54638 deadline: 1680281698054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,059 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:58,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54630 deadline: 1680281698058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,060 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1680281638048, max timestamp=1680281638058Num uniques 859; 2023-03-31 16:53:58,060 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 16:53:58,060 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680281638048, max timestamp=1680281638058Num uniques 859; 2023-03-31 16:53:58,060 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39325, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 16:53:58,060 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1680281638048, max timestamp=1680281638058Num uniques 859; 2023-03-31 16:53:58,061 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 16:53:58,079 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:58,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54628 deadline: 1680281698076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,079 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:58,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54604 deadline: 1680281698078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,081 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:58,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54602 deadline: 1680281698079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,164 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:58,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54638 deadline: 1680281698161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,164 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:58,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54630 deadline: 1680281698162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,184 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:58,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54628 deadline: 1680281698181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,184 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:58,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54604 deadline: 1680281698181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,186 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:58,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54602 deadline: 1680281698183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,238 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=441 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/B/0451d5b5df784582b26a1fdf5a5e50c0 2023-03-31 16:53:58,251 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=441 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/C/59c1ec4d4c5e4d0aa38c098f66178686 2023-03-31 16:53:58,257 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/A/fe6024bfe55a48a29d3eb5e32c7899d8 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/fe6024bfe55a48a29d3eb5e32c7899d8 2023-03-31 16:53:58,262 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/fe6024bfe55a48a29d3eb5e32c7899d8, entries=100, sequenceid=441, filesize=9.5 K 2023-03-31 16:53:58,263 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/B/0451d5b5df784582b26a1fdf5a5e50c0 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/0451d5b5df784582b26a1fdf5a5e50c0 2023-03-31 16:53:58,269 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/0451d5b5df784582b26a1fdf5a5e50c0, entries=100, sequenceid=441, filesize=9.5 K 2023-03-31 16:53:58,270 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/C/59c1ec4d4c5e4d0aa38c098f66178686 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/59c1ec4d4c5e4d0aa38c098f66178686 2023-03-31 16:53:58,276 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/59c1ec4d4c5e4d0aa38c098f66178686, entries=100, sequenceid=441, filesize=9.5 K 2023-03-31 16:53:58,277 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=125.24 KB/128250 for e94df50bb46b68756c6a704332e1c5f4 in 471ms, sequenceid=441, compaction requested=true 2023-03-31 16:53:58,277 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e94df50bb46b68756c6a704332e1c5f4: 2023-03-31 16:53:58,277 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:58,277 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:53:58,277 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:53:58,277 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,277 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:53:58,277 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:53:58,279 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,279 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:58,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:58,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:58,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:58,280 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:58,280 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:53:58,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:58,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:58,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:58,282 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:53:58,282 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b700059[Count = 0] remaining members to acquire global barrier 2023-03-31 16:53:58,282 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:53:58,282 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:58,284 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:58,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:58,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:58,285 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:53:58,285 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:53:58,285 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:53:58,285 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,285 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:53:58,286 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,286 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:53:58,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:58,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:58,286 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:58,286 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:53:58,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:58,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:58,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:58,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:58,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:58,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:53:58,290 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:53:58,290 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:53:58,290 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:53:58,290 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:53:58,290 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:53:58,290 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:53:58,292 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:58,292 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:58,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:58,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:58,292 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:58,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:58,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:58,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:58,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:58,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:58,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:58,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:58,294 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:58,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:58,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:58,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,297 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:58,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:58,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,302 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:58,302 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,302 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:58,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:58,302 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:58,302 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:58,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:58,302 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:53:58,302 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:53:58,302 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:58,302 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,302 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:58,302 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:58,302 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:58,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:58,302 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:53:58,302 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:53:58,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:58,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on e94df50bb46b68756c6a704332e1c5f4 2023-03-31 16:53:58,370 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e94df50bb46b68756c6a704332e1c5f4 3/3 column families, dataSize=131.95 KB heapSize=375.22 KB 2023-03-31 16:53:58,371 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=A 2023-03-31 16:53:58,371 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:53:58,371 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=B 2023-03-31 16:53:58,371 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:53:58,371 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=C 2023-03-31 16:53:58,371 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:53:58,383 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:58,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54630 deadline: 1680281698382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,383 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:58,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54638 deadline: 1680281698383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,386 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:58,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54604 deadline: 1680281698386, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,387 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:58,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54628 deadline: 1680281698386, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,388 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=478 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/A/ae71033970e340c9a65b59dc211107e9 2023-03-31 16:53:58,395 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:58,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54602 deadline: 1680281698388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,402 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:53:58,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:53:58,405 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:53:58,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:53:58,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:53:58,406 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:53:58,406 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:58,407 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:53:58,407 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:53:58,407 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:58,407 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:58,409 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:58,409 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:58,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:58,410 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:58,410 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:53:58,410 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:58,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:53:58,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:58,411 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:53:58,411 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:53:58,411 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:53:58,411 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:53:58,411 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:58,411 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:53:58,416 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:53:58,416 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:53:58,416 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:58,416 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. started... 2023-03-31 16:53:58,416 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. as already flushing 2023-03-31 16:53:58,416 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:58,416 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:53:58,416 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:53:58,416 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,416 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:53:58,416 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:53:58,418 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:58,418 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:58,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:58,419 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:58,419 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:53:58,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:58,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:58,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:58,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:58,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:53:58,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50199060[Count = 0] remaining members to acquire global barrier 2023-03-31 16:53:58,421 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:53:58,421 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:58,422 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:58,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:58,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:58,423 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:53:58,423 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:53:58,423 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:53:58,423 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,423 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:53:58,424 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:53:58,424 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,424 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:58,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:58,424 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:53:58,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:58,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:58,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:58,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:58,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:58,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:58,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:53:58,427 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:53:58,427 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:53:58,427 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:53:58,427 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:53:58,427 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:53:58,427 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:53:58,430 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:58,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:58,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:58,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:58,430 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:58,430 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:58,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:58,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:58,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:58,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:58,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:58,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:58,444 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:58,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:58,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:58,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,446 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:58,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:58,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,454 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,454 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:58,454 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:58,454 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,454 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:58,454 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:58,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:58,454 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:58,454 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:58,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:58,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:53:58,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:53:58,454 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:58,454 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:58,455 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:53:58,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:58,455 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:53:58,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:58,486 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:58,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54630 deadline: 1680281698484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,486 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:58,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54638 deadline: 1680281698484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,555 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:53:58,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:53:58,559 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:53:58,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:53:58,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:53:58,560 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:53:58,560 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:58,560 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:53:58,560 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:53:58,561 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:58,561 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:58,562 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,562 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:58,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:58,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:58,563 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,563 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:53:58,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:58,563 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:58,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:53:58,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:58,564 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:53:58,564 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:53:58,564 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:53:58,564 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:53:58,564 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:58,564 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:53:58,565 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:53:58,565 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:53:58,565 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:58,565 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. started... 2023-03-31 16:53:58,565 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. as already flushing 2023-03-31 16:53:58,565 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:58,565 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:53:58,565 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:53:58,565 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,565 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:53:58,565 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:53:58,567 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,567 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:58,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:58,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:58,567 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:58,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:58,567 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:53:58,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:58,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:58,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:58,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:53:58,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7de04ef1[Count = 0] remaining members to acquire global barrier 2023-03-31 16:53:58,569 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:53:58,569 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:58,573 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:58,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:58,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:58,574 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:53:58,574 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:53:58,574 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:53:58,574 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,574 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:53:58,575 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:53:58,576 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:58,576 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:53:58,576 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:58,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:58,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:58,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:58,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:58,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:58,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:58,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:53:58,579 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:53:58,579 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:53:58,579 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:53:58,579 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:53:58,579 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:53:58,579 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:53:58,580 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:58,580 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:58,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:58,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:58,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:58,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:58,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:58,580 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:58,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:58,581 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:58,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:58,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:58,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:58,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:58,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:58,583 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:58,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:58,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,585 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,585 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:58,585 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:58,585 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:53:58,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:53:58,585 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:58,585 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:58,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:58,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:58,586 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:53:58,585 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:58,586 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:53:58,586 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:58,585 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:58,586 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:58,586 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:53:58,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:58,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:58,686 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:53:58,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:53:58,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:53:58,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:53:58,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:53:58,690 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:53:58,690 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:58,690 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:53:58,690 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:53:58,690 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:58,690 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:58,692 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:58,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54630 deadline: 1680281698687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,692 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:58,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54604 deadline: 1680281698687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,693 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:58,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54638 deadline: 1680281698687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,693 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:58,693 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54628 deadline: 1680281698688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,693 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:58,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:58,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:58,693 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,694 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:53:58,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:58,694 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:58,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:53:58,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:58,694 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:53:58,695 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:53:58,695 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:53:58,695 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:53:58,695 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:58,695 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:53:58,695 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:53:58,695 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:53:58,695 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:58,695 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. started... 2023-03-31 16:53:58,695 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. as already flushing 2023-03-31 16:53:58,695 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:58,695 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:53:58,695 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:53:58,695 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,696 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:53:58,696 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:53:58,697 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:58,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54602 deadline: 1680281698696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,697 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:58,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:58,698 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:58,698 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:58,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:58,698 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:53:58,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:58,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:58,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:58,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:53:58,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a078c6b[Count = 0] remaining members to acquire global barrier 2023-03-31 16:53:58,700 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:53:58,700 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:58,701 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:58,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:58,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:58,702 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:53:58,702 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:53:58,702 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:53:58,702 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,702 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:53:58,703 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,703 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:53:58,703 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:58,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:58,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:58,703 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:53:58,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:58,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:58,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:58,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:58,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:58,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:53:58,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:53:58,708 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:53:58,708 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:53:58,708 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:53:58,708 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:53:58,708 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:53:58,711 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:58,711 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:58,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:58,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:58,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:58,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:58,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:58,711 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:58,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:58,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:58,712 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:58,713 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:58,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:58,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:58,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:58,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,716 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:58,716 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:58,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:58,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:58,717 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,717 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:58,717 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:58,717 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:58,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:58,717 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:53:58,717 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:53:58,717 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,717 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:58,717 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:58,717 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:58,718 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:53:58,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:58,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:58,718 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:53:58,718 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:53:58,806 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=478 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/B/890edee61afc4cc181ba124e76a6f665 2023-03-31 16:53:58,818 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:53:58,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:53:58,819 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.74 KB at sequenceid=478 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/C/13fdc9711b9f4c28bba6cd4307431710 2023-03-31 16:53:58,821 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:53:58,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:53:58,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:53:58,823 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:53:58,823 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:58,823 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:53:58,823 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:53:58,824 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:58,824 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:58,826 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,826 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/A/ae71033970e340c9a65b59dc211107e9 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/ae71033970e340c9a65b59dc211107e9 2023-03-31 16:53:58,826 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:58,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:58,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:58,826 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,826 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:53:58,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:58,827 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:58,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:53:58,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:58,827 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:53:58,827 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:53:58,827 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:53:58,828 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:53:58,828 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:58,828 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:53:58,828 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:53:58,828 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:53:58,828 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:58,828 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. started... 2023-03-31 16:53:58,828 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. as already flushing 2023-03-31 16:53:58,828 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:58,828 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:53:58,828 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:53:58,828 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,828 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:53:58,828 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:53:58,830 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:58,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:58,830 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:58,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:58,831 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:58,831 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:53:58,831 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/ae71033970e340c9a65b59dc211107e9, entries=200, sequenceid=478, filesize=14.3 K 2023-03-31 16:53:58,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:58,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:58,832 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/B/890edee61afc4cc181ba124e76a6f665 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/890edee61afc4cc181ba124e76a6f665 2023-03-31 16:53:58,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:58,833 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:53:58,833 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61ce2dba[Count = 0] remaining members to acquire global barrier 2023-03-31 16:53:58,833 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:53:58,833 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:58,836 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,836 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:53:58,838 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/890edee61afc4cc181ba124e76a6f665, entries=150, sequenceid=478, filesize=11.9 K 2023-03-31 16:53:58,839 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/C/13fdc9711b9f4c28bba6cd4307431710 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/13fdc9711b9f4c28bba6cd4307431710 2023-03-31 16:53:58,844 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:58,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:58,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:58,844 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:53:58,844 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:53:58,844 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:53:58,844 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/13fdc9711b9f4c28bba6cd4307431710, entries=150, sequenceid=478, filesize=11.9 K 2023-03-31 16:53:58,845 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~138.66 KB/141990, heapSize ~395.09 KB/404576, currentSize=40.25 KB/41220 for e94df50bb46b68756c6a704332e1c5f4 in 475ms, sequenceid=478, compaction requested=true 2023-03-31 16:53:58,845 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e94df50bb46b68756c6a704332e1c5f4: 2023-03-31 16:53:58,845 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:53:58,845 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 16:53:58,845 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 16:53:58,845 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:53:58,846 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:53:58,846 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,846 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:53:58,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:58,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:58,846 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:58,847 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:53:58,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:58,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:58,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:58,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:58,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:58,849 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49456 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 16:53:58,849 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47016 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 16:53:58,849 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1912): e94df50bb46b68756c6a704332e1c5f4/A is initiating minor compaction (all files) 2023-03-31 16:53:58,849 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): e94df50bb46b68756c6a704332e1c5f4/B is initiating minor compaction (all files) 2023-03-31 16:53:58,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,849 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e94df50bb46b68756c6a704332e1c5f4/B in TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:58,849 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e94df50bb46b68756c6a704332e1c5f4/A in TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:58,849 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/22eeaae46aaa4aedac30fc05e2a1fcdd, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/d6a82a8c0157451eb9050e6a57525e2b, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/0451d5b5df784582b26a1fdf5a5e50c0, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/890edee61afc4cc181ba124e76a6f665] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp, totalSize=45.9 K 2023-03-31 16:53:58,849 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/940492fd0c174e0286672200d5b9b2cd, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/221e1ab33201446c9815b31c933a0f71, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/fe6024bfe55a48a29d3eb5e32c7899d8, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/ae71033970e340c9a65b59dc211107e9] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp, totalSize=48.3 K 2023-03-31 16:53:58,849 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 22eeaae46aaa4aedac30fc05e2a1fcdd, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=391, earliestPutTs=1680281636147 2023-03-31 16:53:58,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:53:58,850 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:53:58,850 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 940492fd0c174e0286672200d5b9b2cd, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=391, earliestPutTs=1680281636147 2023-03-31 16:53:58,850 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:53:58,850 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:53:58,850 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:53:58,850 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting d6a82a8c0157451eb9050e6a57525e2b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=427, earliestPutTs=1680281636248 2023-03-31 16:53:58,850 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:53:58,850 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:53:58,850 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 221e1ab33201446c9815b31c933a0f71, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=427, earliestPutTs=1680281636248 2023-03-31 16:53:58,851 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 0451d5b5df784582b26a1fdf5a5e50c0, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=441, earliestPutTs=1680281636909 2023-03-31 16:53:58,851 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting fe6024bfe55a48a29d3eb5e32c7899d8, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=441, earliestPutTs=1680281636909 2023-03-31 16:53:58,851 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 890edee61afc4cc181ba124e76a6f665, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=478, earliestPutTs=1680281638074 2023-03-31 16:53:58,852 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting ae71033970e340c9a65b59dc211107e9, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=478, earliestPutTs=1680281638074 2023-03-31 16:53:58,856 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:58,856 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:58,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:58,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:58,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:58,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:58,857 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:58,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:58,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:58,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:58,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:58,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:58,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:58,862 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:58,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:58,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:58,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:58,865 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,872 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,872 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:58,872 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:58,872 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:58,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:58,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:58,872 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,872 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:58,872 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:58,872 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:58,872 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:58,873 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:58,873 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:53:58,873 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:53:58,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:58,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:58,873 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:53:58,873 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:53:58,877 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] throttle.PressureAwareThroughputController(145): e94df50bb46b68756c6a704332e1c5f4#A#compaction#75 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:53:58,879 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): e94df50bb46b68756c6a704332e1c5f4#B#compaction#76 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:53:58,974 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:53:58,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:53:58,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:53:58,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:53:58,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:53:58,979 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:53:58,979 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:58,979 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:53:58,979 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:53:58,979 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:58,979 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:58,980 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:58,980 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:58,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:58,981 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:58,981 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:53:58,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:58,981 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:58,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:53:58,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:58,982 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:53:58,982 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:53:58,982 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:53:58,982 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:53:58,982 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:58,982 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:53:58,991 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:53:58,991 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:58,991 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:53:58,992 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. started... 2023-03-31 16:53:58,992 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e94df50bb46b68756c6a704332e1c5f4 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-03-31 16:53:58,992 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=A 2023-03-31 16:53:58,992 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:53:58,992 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=B 2023-03-31 16:53:58,992 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:53:58,992 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=C 2023-03-31 16:53:58,992 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:53:59,003 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=487 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/A/8275bd19017d49fc90234317853bf5a1 2023-03-31 16:53:59,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on e94df50bb46b68756c6a704332e1c5f4 2023-03-31 16:53:59,005 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. as already flushing 2023-03-31 16:53:59,043 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=487 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/B/fd2db7d75aca4434ae6b574dbbb80a07 2023-03-31 16:53:59,046 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680281639033, max timestamp=1680281639044Num uniques 859; 2023-03-31 16:53:59,046 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680281639033, max timestamp=1680281639044Num uniques 859; 2023-03-31 16:53:59,047 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 16:53:59,047 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 16:53:59,047 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680281639033, max timestamp=1680281639044Num uniques 859; 2023-03-31 16:53:59,048 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 16:53:59,076 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=487 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/C/62602353da6e4ada80784ed765fbd94a 2023-03-31 16:53:59,087 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/A/8275bd19017d49fc90234317853bf5a1 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/8275bd19017d49fc90234317853bf5a1 2023-03-31 16:53:59,095 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:59,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54638 deadline: 1680281699090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,099 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:59,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54630 deadline: 1680281699094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,105 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/8275bd19017d49fc90234317853bf5a1, entries=100, sequenceid=487, filesize=9.5 K 2023-03-31 16:53:59,106 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/B/fd2db7d75aca4434ae6b574dbbb80a07 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/fd2db7d75aca4434ae6b574dbbb80a07 2023-03-31 16:53:59,121 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/fd2db7d75aca4434ae6b574dbbb80a07, entries=100, sequenceid=487, filesize=9.5 K 2023-03-31 16:53:59,123 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/C/62602353da6e4ada80784ed765fbd94a as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/62602353da6e4ada80784ed765fbd94a 2023-03-31 16:53:59,133 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/62602353da6e4ada80784ed765fbd94a, entries=100, sequenceid=487, filesize=9.5 K 2023-03-31 16:53:59,134 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=138.65 KB/141980 for e94df50bb46b68756c6a704332e1c5f4 in 142ms, sequenceid=487, compaction requested=true 2023-03-31 16:53:59,134 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e94df50bb46b68756c6a704332e1c5f4: 2023-03-31 16:53:59,134 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:59,134 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:53:59,134 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:53:59,134 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,136 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:53:59,136 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:53:59,138 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,138 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:59,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:59,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:59,138 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:59,138 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:53:59,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:59,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:59,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:59,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:59,142 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:53:59,142 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7dc82b73[Count = 0] remaining members to acquire global barrier 2023-03-31 16:53:59,142 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:53:59,142 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:59,148 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:59,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:59,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:59,148 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:53:59,148 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:53:59,148 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:53:59,149 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,149 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:53:59,150 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:59,150 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:53:59,150 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:53:59,150 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:53:59,150 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:53:59,150 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:53:59,150 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:53:59,150 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:53:59,150 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:53:59,152 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:59,152 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:59,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:59,152 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:59,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:59,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:59,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:59,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:59,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:59,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:59,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:59,153 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:59,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:59,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:59,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:59,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,154 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:59,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:59,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,157 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,157 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:59,157 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:59,157 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,158 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:59,158 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:59,158 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:59,158 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:59,158 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:59,158 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:59,158 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:53:59,158 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:53:59,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:59,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:59,158 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:53:59,158 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:53:59,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:59,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:59,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on e94df50bb46b68756c6a704332e1c5f4 2023-03-31 16:53:59,201 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e94df50bb46b68756c6a704332e1c5f4 3/3 column families, dataSize=152.07 KB heapSize=435.69 KB 2023-03-31 16:53:59,201 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=A 2023-03-31 16:53:59,201 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-31 16:53:59,201 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=B 2023-03-31 16:53:59,201 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-31 16:53:59,201 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=C 2023-03-31 16:53:59,201 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-31 16:53:59,206 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:59,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54638 deadline: 1680281699203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,210 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:59,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54602 deadline: 1680281699205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,210 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:59,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54630 deadline: 1680281699207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,213 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:59,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54628 deadline: 1680281699212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,214 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:59,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54604 deadline: 1680281699212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,230 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=526 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/A/78b890b2565c46888f2b5832bf7cd266 2023-03-31 16:53:59,258 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:53:59,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:53:59,262 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:53:59,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:53:59,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:53:59,272 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:53:59,272 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:59,272 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:53:59,272 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:53:59,273 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:59,273 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:59,275 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:59,275 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:59,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:59,277 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:59,277 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:53:59,277 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:59,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:53:59,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:59,277 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:53:59,282 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:53:59,282 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:53:59,282 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:53:59,282 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:59,282 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:53:59,288 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:53:59,288 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:53:59,288 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:59,288 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. started... 2023-03-31 16:53:59,288 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. as already flushing 2023-03-31 16:53:59,288 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:59,288 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:53:59,288 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:53:59,288 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,288 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:53:59,288 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:53:59,290 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:59,291 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:59,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:59,291 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:59,291 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:53:59,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:59,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:59,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:59,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:59,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:53:59,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22065fd[Count = 0] remaining members to acquire global barrier 2023-03-31 16:53:59,294 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:53:59,294 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:59,296 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:59,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:59,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:59,296 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:53:59,296 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:53:59,296 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:53:59,296 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,296 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:53:59,297 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:53:59,297 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:59,297 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:53:59,298 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:59,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:59,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:59,299 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/B/2d3c5617a87941a6a8b17087da7dcd5d as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/2d3c5617a87941a6a8b17087da7dcd5d 2023-03-31 16:53:59,299 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/A/d16eb543a1154647986050637dac1131 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/d16eb543a1154647986050637dac1131 2023-03-31 16:53:59,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:59,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:59,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:59,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:59,311 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:59,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54638 deadline: 1680281699308, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,315 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e94df50bb46b68756c6a704332e1c5f4/B of e94df50bb46b68756c6a704332e1c5f4 into 2d3c5617a87941a6a8b17087da7dcd5d(size=12.7 K), total size for store is 22.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:53:59,315 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e94df50bb46b68756c6a704332e1c5f4: 2023-03-31 16:53:59,315 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4., storeName=e94df50bb46b68756c6a704332e1c5f4/B, priority=12, startTime=1680281638845; duration=0sec 2023-03-31 16:53:59,318 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:59,317 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:59,316 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e94df50bb46b68756c6a704332e1c5f4/A of e94df50bb46b68756c6a704332e1c5f4 into d16eb543a1154647986050637dac1131(size=12.7 K), total size for store is 22.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:53:59,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54602 deadline: 1680281699311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,323 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e94df50bb46b68756c6a704332e1c5f4: 2023-03-31 16:53:59,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:53:59,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54630 deadline: 1680281699312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:53:59,324 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:53:59,324 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:53:59,324 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:53:59,324 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:59,323 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:53:59,324 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-31 16:53:59,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54628 deadline: 1680281699315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,324 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:53:59,324 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:59,323 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4., storeName=e94df50bb46b68756c6a704332e1c5f4/A, priority=12, startTime=1680281638845; duration=0sec 2023-03-31 16:53:59,324 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:53:59,325 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:53:59,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54604 deadline: 1680281699316, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,327 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 56758 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-31 16:53:59,327 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): e94df50bb46b68756c6a704332e1c5f4/C is initiating minor compaction (all files) 2023-03-31 16:53:59,327 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e94df50bb46b68756c6a704332e1c5f4/C in TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:59,327 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/1c85f91970d54826961dc4d064feb5a4, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/01bb121ed8c048b99bb20915f4faf16d, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/59c1ec4d4c5e4d0aa38c098f66178686, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/13fdc9711b9f4c28bba6cd4307431710, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/62602353da6e4ada80784ed765fbd94a] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp, totalSize=55.4 K 2023-03-31 16:53:59,328 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 1c85f91970d54826961dc4d064feb5a4, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=391, earliestPutTs=1680281636147 2023-03-31 16:53:59,328 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 01bb121ed8c048b99bb20915f4faf16d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=427, earliestPutTs=1680281636248 2023-03-31 16:53:59,329 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 59c1ec4d4c5e4d0aa38c098f66178686, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=441, earliestPutTs=1680281636909 2023-03-31 16:53:59,329 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 13fdc9711b9f4c28bba6cd4307431710, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=478, earliestPutTs=1680281638074 2023-03-31 16:53:59,329 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 62602353da6e4ada80784ed765fbd94a, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=487, earliestPutTs=1680281638378 2023-03-31 16:53:59,331 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:59,331 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:59,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:59,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:59,331 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:59,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:59,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:59,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:59,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:59,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:59,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:59,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:59,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:59,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:59,333 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:59,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:59,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:59,335 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,339 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:59,339 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:59,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:59,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:59,339 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,339 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:59,339 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:59,339 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,339 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:59,339 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:59,339 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:59,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:59,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:59,340 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:59,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:53:59,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:53:59,340 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:53:59,340 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:53:59,347 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): e94df50bb46b68756c6a704332e1c5f4#C#compaction#82 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:53:59,440 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:53:59,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:53:59,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:53:59,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:53:59,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:53:59,447 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:53:59,447 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:59,447 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:53:59,447 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:53:59,448 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:59,448 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:59,449 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:59,449 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:59,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:59,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:59,450 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,450 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:53:59,450 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:59,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:53:59,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:59,451 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:53:59,451 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:53:59,451 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:53:59,451 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:53:59,451 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:59,451 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:53:59,452 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:53:59,452 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:53:59,452 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:59,452 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. started... 2023-03-31 16:53:59,452 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. as already flushing 2023-03-31 16:53:59,452 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:59,452 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:53:59,452 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:53:59,452 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,452 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:53:59,452 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:53:59,454 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,454 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:59,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:59,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:59,454 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:59,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:59,454 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:53:59,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:59,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:59,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:59,465 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:53:59,465 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64a2c506[Count = 0] remaining members to acquire global barrier 2023-03-31 16:53:59,465 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:53:59,466 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:59,468 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:59,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:59,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:59,468 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:53:59,468 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:53:59,468 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:53:59,469 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,469 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:53:59,470 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,470 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:53:59,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:59,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:59,470 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:59,470 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:53:59,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:59,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:59,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:59,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:59,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:59,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:53:59,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:53:59,475 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:53:59,475 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:53:59,475 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:53:59,475 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:53:59,475 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:53:59,478 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:59,479 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:59,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:59,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:59,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:59,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:59,479 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:59,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:59,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:59,479 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:59,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:59,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:59,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:59,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:59,481 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:59,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:59,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:59,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,484 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:59,485 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:59,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:59,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:59,485 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,485 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:59,485 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:59,485 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:53:59,485 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:53:59,485 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:59,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:59,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:59,485 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,485 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:59,486 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:53:59,486 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:53:59,486 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:59,486 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:59,486 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:53:59,514 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:59,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54638 deadline: 1680281699513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,526 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:59,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54602 deadline: 1680281699525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,527 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:59,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54630 deadline: 1680281699525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,528 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:59,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54628 deadline: 1680281699526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,528 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:59,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54604 deadline: 1680281699526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,586 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:53:59,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:53:59,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:53:59,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:53:59,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:53:59,589 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:53:59,589 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:59,589 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:53:59,590 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:53:59,590 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:59,590 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:59,595 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:59,595 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:59,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:59,595 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,595 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:53:59,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:59,595 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:59,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:53:59,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:59,596 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:53:59,596 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:53:59,596 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:53:59,596 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:53:59,596 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:59,597 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:53:59,597 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:53:59,597 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:53:59,597 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:59,597 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. started... 2023-03-31 16:53:59,597 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. as already flushing 2023-03-31 16:53:59,597 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:59,598 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:53:59,598 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:53:59,598 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,598 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:53:59,598 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:53:59,599 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,599 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:59,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:59,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:59,600 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:59,600 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:53:59,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:59,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:59,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:59,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:59,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:53:59,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@741197d5[Count = 0] remaining members to acquire global barrier 2023-03-31 16:53:59,601 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:53:59,601 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:59,603 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:59,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:59,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:59,603 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:53:59,603 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:53:59,603 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,603 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:53:59,603 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:53:59,605 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:59,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:59,605 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:53:59,606 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:59,606 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:53:59,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:59,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:59,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:59,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:59,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:59,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:53:59,611 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:53:59,611 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:53:59,611 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:53:59,611 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:53:59,611 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:53:59,611 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:53:59,615 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:59,615 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:59,615 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:59,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:59,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:59,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:59,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:59,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:59,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:59,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:59,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:59,616 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:59,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:59,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:59,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:59,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,618 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:59,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:59,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,620 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:59,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:59,620 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:59,620 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:59,620 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:59,620 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:53:59,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:53:59,620 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:59,621 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:59,621 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:53:59,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:59,621 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:53:59,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:59,621 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:53:59,621 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,621 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:59,621 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:59,621 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:59,658 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=526 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/B/ac5fd7c80e094d228f191380eb478244 2023-03-31 16:53:59,721 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:53:59,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:53:59,725 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:53:59,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:53:59,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:53:59,736 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:53:59,736 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:59,736 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:53:59,736 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:53:59,737 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:59,737 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:59,739 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:59,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:59,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:59,739 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:59,741 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:59,741 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,741 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:53:59,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:53:59,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:59,741 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:53:59,742 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:53:59,742 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:53:59,742 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:53:59,742 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:59,742 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:53:59,743 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:53:59,743 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:53:59,743 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:59,743 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. started... 2023-03-31 16:53:59,743 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. as already flushing 2023-03-31 16:53:59,743 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:59,743 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:53:59,743 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:53:59,743 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,743 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:53:59,743 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:53:59,750 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:59,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:59,751 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:59,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:59,751 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:59,751 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:53:59,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:59,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:59,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:59,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:53:59,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@483eb997[Count = 0] remaining members to acquire global barrier 2023-03-31 16:53:59,754 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:53:59,754 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:59,757 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:59,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:59,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:59,757 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:53:59,757 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:53:59,757 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:53:59,758 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,758 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:53:59,760 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:59,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:59,778 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/C/55d8160c4f974b8cb369c3aa22d21c9b as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/55d8160c4f974b8cb369c3aa22d21c9b 2023-03-31 16:53:59,760 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:53:59,779 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:59,779 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:53:59,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:59,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:59,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:59,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:59,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:59,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:53:59,786 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:53:59,786 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:53:59,786 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:53:59,786 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:53:59,786 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:53:59,786 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:53:59,787 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in e94df50bb46b68756c6a704332e1c5f4/C of e94df50bb46b68756c6a704332e1c5f4 into 55d8160c4f974b8cb369c3aa22d21c9b(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:53:59,787 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e94df50bb46b68756c6a704332e1c5f4: 2023-03-31 16:53:59,787 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4., storeName=e94df50bb46b68756c6a704332e1c5f4/C, priority=11, startTime=1680281638846; duration=0sec 2023-03-31 16:53:59,787 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:53:59,792 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:59,792 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:59,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:59,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:59,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:59,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:59,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:59,793 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:59,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:59,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:59,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:59,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:59,794 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:59,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:59,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:59,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,795 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:59,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:59,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,799 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,799 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:59,799 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:59,799 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,799 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:59,799 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:59,799 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:59,799 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:59,799 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:59,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:53:59,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:53:59,799 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:59,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:59,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:59,799 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:53:59,799 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:53:59,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:59,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:59,818 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:59,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54638 deadline: 1680281699817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,828 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:59,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54602 deadline: 1680281699828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,834 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:59,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54628 deadline: 1680281699829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,834 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:59,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54604 deadline: 1680281699829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,834 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:53:59,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54630 deadline: 1680281699830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,900 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:53:59,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:53:59,902 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:53:59,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:53:59,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:53:59,903 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:53:59,903 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:59,903 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:53:59,903 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:53:59,903 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:59,904 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:59,905 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,905 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:59,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:59,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:59,905 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,905 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:53:59,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:59,906 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:59,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:53:59,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:59,906 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:53:59,906 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:53:59,906 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:53:59,906 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:53:59,907 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:53:59,907 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:53:59,907 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:53:59,907 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:53:59,907 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:59,907 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. started... 2023-03-31 16:53:59,907 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. as already flushing 2023-03-31 16:53:59,907 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:53:59,907 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:53:59,907 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:53:59,907 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,907 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:53:59,907 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:53:59,909 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,909 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:59,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:59,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:59,910 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:59,910 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:53:59,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:59,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:59,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:59,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:59,911 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:53:59,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49b022e0[Count = 0] remaining members to acquire global barrier 2023-03-31 16:53:59,912 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:53:59,912 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:59,913 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:59,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:59,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:59,913 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:53:59,913 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:53:59,913 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:53:59,914 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,914 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:53:59,916 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,916 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:53:59,916 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:59,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:59,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:59,916 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:53:59,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:59,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:59,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:59,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:59,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:59,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:53:59,919 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:53:59,920 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:53:59,920 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:53:59,920 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:53:59,920 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:53:59,920 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:53:59,921 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:59,921 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:59,921 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:59,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:59,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:59,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:59,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:53:59,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:53:59,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:59,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:53:59,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:53:59,922 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:59,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:59,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:53:59,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:59,924 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:53:59,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:53:59,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,927 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,927 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:59,927 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:53:59,927 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:53:59,927 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:53:59,927 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:53:59,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:53:59,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:53:59,927 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:53:59,927 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:53:59,927 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:53:59,927 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:59,927 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:53:59,927 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:53:59,927 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:53:59,927 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:53:59,928 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:53:59,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:53:59,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:00,028 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:00,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:00,030 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:00,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:00,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:00,031 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:00,031 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:00,031 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:00,031 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:00,031 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:00,031 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:00,033 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:00,033 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:00,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:00,033 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,033 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:00,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:00,034 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:00,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:00,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:00,035 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:00,035 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:00,035 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:00,035 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:00,035 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:00,035 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:00,036 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:00,036 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:00,036 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:54:00,036 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. started... 2023-03-31 16:54:00,036 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. as already flushing 2023-03-31 16:54:00,036 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:54:00,036 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:00,036 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:00,036 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,036 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:00,036 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:00,038 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,038 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:00,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:00,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:00,039 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:00,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:00,039 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:00,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:00,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:00,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:00,040 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:00,040 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:00,040 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b485425[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:00,040 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:00,044 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:00,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:00,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:00,045 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:00,045 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:00,045 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,045 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:00,045 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:00,046 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,046 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:00,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:00,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:00,046 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:00,046 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:00,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:00,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:00,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:00,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:00,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:00,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:00,051 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:00,051 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:00,051 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:00,051 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:00,051 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:00,051 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:00,053 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:00,053 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:00,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:00,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:00,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:00,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:00,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:00,053 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:00,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:00,054 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:00,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:00,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:00,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:00,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:00,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:00,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,055 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:00,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:00,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,058 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,058 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:00,058 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:00,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:00,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:00,058 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:00,058 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:00,058 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:00,058 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:00,058 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:00,058 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,058 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:00,058 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:00,058 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:00,058 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:00,059 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:00,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:00,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:00,080 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=526 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/C/bea34f5bbb6a44d28063f8259b6a826d 2023-03-31 16:54:00,086 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/A/78b890b2565c46888f2b5832bf7cd266 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/78b890b2565c46888f2b5832bf7cd266 2023-03-31 16:54:00,091 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/78b890b2565c46888f2b5832bf7cd266, entries=250, sequenceid=526, filesize=16.7 K 2023-03-31 16:54:00,092 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/B/ac5fd7c80e094d228f191380eb478244 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/ac5fd7c80e094d228f191380eb478244 2023-03-31 16:54:00,098 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/ac5fd7c80e094d228f191380eb478244, entries=150, sequenceid=526, filesize=11.9 K 2023-03-31 16:54:00,099 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/C/bea34f5bbb6a44d28063f8259b6a826d as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/bea34f5bbb6a44d28063f8259b6a826d 2023-03-31 16:54:00,104 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/bea34f5bbb6a44d28063f8259b6a826d, entries=150, sequenceid=526, filesize=11.9 K 2023-03-31 16:54:00,104 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~152.07 KB/155720, heapSize ~435.64 KB/446096, currentSize=33.54 KB/34350 for e94df50bb46b68756c6a704332e1c5f4 in 904ms, sequenceid=526, compaction requested=true 2023-03-31 16:54:00,105 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e94df50bb46b68756c6a704332e1c5f4: 2023-03-31 16:54:00,105 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:54:00,105 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 16:54:00,105 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 16:54:00,105 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:54:00,105 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:54:00,113 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39846 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 16:54:00,113 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1912): e94df50bb46b68756c6a704332e1c5f4/A is initiating minor compaction (all files) 2023-03-31 16:54:00,113 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e94df50bb46b68756c6a704332e1c5f4/A in TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:54:00,114 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/d16eb543a1154647986050637dac1131, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/8275bd19017d49fc90234317853bf5a1, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/78b890b2565c46888f2b5832bf7cd266] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp, totalSize=38.9 K 2023-03-31 16:54:00,114 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34966 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 16:54:00,114 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): e94df50bb46b68756c6a704332e1c5f4/B is initiating minor compaction (all files) 2023-03-31 16:54:00,114 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting d16eb543a1154647986050637dac1131, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=478, earliestPutTs=1680281638074 2023-03-31 16:54:00,114 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e94df50bb46b68756c6a704332e1c5f4/B in TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:54:00,114 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/2d3c5617a87941a6a8b17087da7dcd5d, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/fd2db7d75aca4434ae6b574dbbb80a07, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/ac5fd7c80e094d228f191380eb478244] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp, totalSize=34.1 K 2023-03-31 16:54:00,115 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 8275bd19017d49fc90234317853bf5a1, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=487, earliestPutTs=1680281638378 2023-03-31 16:54:00,115 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 2d3c5617a87941a6a8b17087da7dcd5d, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=478, earliestPutTs=1680281638074 2023-03-31 16:54:00,115 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 78b890b2565c46888f2b5832bf7cd266, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=526, earliestPutTs=1680281639086 2023-03-31 16:54:00,115 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting fd2db7d75aca4434ae6b574dbbb80a07, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=487, earliestPutTs=1680281638378 2023-03-31 16:54:00,116 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting ac5fd7c80e094d228f191380eb478244, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=526, earliestPutTs=1680281639088 2023-03-31 16:54:00,126 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] throttle.PressureAwareThroughputController(145): e94df50bb46b68756c6a704332e1c5f4#A#compaction#84 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:54:00,128 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): e94df50bb46b68756c6a704332e1c5f4#B#compaction#85 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:54:00,159 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:00,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:00,161 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:00,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:00,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:00,162 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:00,162 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:00,162 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:00,162 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:00,163 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:00,163 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:00,164 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:00,164 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:00,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:00,164 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,164 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:00,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:00,165 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:00,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:00,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:00,166 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:00,167 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:00,167 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:00,167 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:00,167 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:00,167 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:00,168 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:00,168 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:54:00,168 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:00,168 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. started... 2023-03-31 16:54:00,168 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e94df50bb46b68756c6a704332e1c5f4 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-03-31 16:54:00,169 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=A 2023-03-31 16:54:00,169 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:00,169 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=B 2023-03-31 16:54:00,169 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:00,169 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=C 2023-03-31 16:54:00,169 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:00,180 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=537 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/A/8e110553cd8040b392e868d1ba25330d 2023-03-31 16:54:00,210 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=537 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/B/b2cde8036b11485d86cfbdf0994d8025 2023-03-31 16:54:00,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on e94df50bb46b68756c6a704332e1c5f4 2023-03-31 16:54:00,325 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. as already flushing 2023-03-31 16:54:00,361 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:00,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54638 deadline: 1680281700355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,364 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:00,365 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:00,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54628 deadline: 1680281700359, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54602 deadline: 1680281700359, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,365 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:00,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54604 deadline: 1680281700361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,366 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:00,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54630 deadline: 1680281700361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,467 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:00,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54638 deadline: 1680281700464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,469 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:00,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54628 deadline: 1680281700467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,469 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:00,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54602 deadline: 1680281700467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,469 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:00,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54630 deadline: 1680281700467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,471 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:00,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54604 deadline: 1680281700468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,554 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/A/cf90fdeba90b46b6862e8535e4ad999a as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/cf90fdeba90b46b6862e8535e4ad999a 2023-03-31 16:54:00,561 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e94df50bb46b68756c6a704332e1c5f4/A of e94df50bb46b68756c6a704332e1c5f4 into cf90fdeba90b46b6862e8535e4ad999a(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:54:00,561 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e94df50bb46b68756c6a704332e1c5f4: 2023-03-31 16:54:00,561 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4., storeName=e94df50bb46b68756c6a704332e1c5f4/A, priority=13, startTime=1680281640105; duration=0sec 2023-03-31 16:54:00,561 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:54:00,561 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-03-31 16:54:00,562 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-31 16:54:00,562 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-31 16:54:00,562 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. because compaction request was cancelled 2023-03-31 16:54:00,565 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/B/a55c2c3809b14425929ae2608a09ea40 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/a55c2c3809b14425929ae2608a09ea40 2023-03-31 16:54:00,571 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e94df50bb46b68756c6a704332e1c5f4/B of e94df50bb46b68756c6a704332e1c5f4 into a55c2c3809b14425929ae2608a09ea40(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:54:00,571 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e94df50bb46b68756c6a704332e1c5f4: 2023-03-31 16:54:00,571 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4., storeName=e94df50bb46b68756c6a704332e1c5f4/B, priority=13, startTime=1680281640105; duration=0sec 2023-03-31 16:54:00,572 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:54:00,634 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=537 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/C/d1eaaaf307824034a64d551410fcc92f 2023-03-31 16:54:00,640 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/A/8e110553cd8040b392e868d1ba25330d as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/8e110553cd8040b392e868d1ba25330d 2023-03-31 16:54:00,647 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/8e110553cd8040b392e868d1ba25330d, entries=150, sequenceid=537, filesize=11.9 K 2023-03-31 16:54:00,648 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/B/b2cde8036b11485d86cfbdf0994d8025 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/b2cde8036b11485d86cfbdf0994d8025 2023-03-31 16:54:00,653 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/b2cde8036b11485d86cfbdf0994d8025, entries=150, sequenceid=537, filesize=11.9 K 2023-03-31 16:54:00,654 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/C/d1eaaaf307824034a64d551410fcc92f as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/d1eaaaf307824034a64d551410fcc92f 2023-03-31 16:54:00,658 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/d1eaaaf307824034a64d551410fcc92f, entries=150, sequenceid=537, filesize=11.9 K 2023-03-31 16:54:00,659 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=147.60 KB/151140 for e94df50bb46b68756c6a704332e1c5f4 in 491ms, sequenceid=537, compaction requested=true 2023-03-31 16:54:00,659 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e94df50bb46b68756c6a704332e1c5f4: 2023-03-31 16:54:00,660 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:54:00,660 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:00,660 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:00,660 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,660 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:00,660 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:00,661 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,661 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:00,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:00,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:00,662 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:00,662 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:00,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:00,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:00,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:00,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:00,663 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:00,663 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@431fc3a6[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:00,663 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:00,663 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:00,665 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:00,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:00,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:00,665 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:00,665 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:00,665 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:00,665 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,665 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:00,666 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,666 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:00,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:00,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:00,666 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:00,666 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:00,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:00,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:00,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:00,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:00,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:00,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:00,669 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:00,669 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:00,669 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:00,669 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:00,669 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:00,669 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:00,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on e94df50bb46b68756c6a704332e1c5f4 2023-03-31 16:54:00,671 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e94df50bb46b68756c6a704332e1c5f4 3/3 column families, dataSize=154.31 KB heapSize=459.66 KB 2023-03-31 16:54:00,671 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=A 2023-03-31 16:54:00,671 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:54:00,671 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=B 2023-03-31 16:54:00,671 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:00,671 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:54:00,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:00,671 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=C 2023-03-31 16:54:00,671 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:00,671 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:54:00,672 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:00,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:00,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:00,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:00,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:00,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:00,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:00,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:00,673 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:00,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:00,676 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:00,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54604 deadline: 1680281700673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:00,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:00,678 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:00,678 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:00,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54602 deadline: 1680281700675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,679 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:00,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54630 deadline: 1680281700676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,679 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:00,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54628 deadline: 1680281700677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,680 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:00,680 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:00,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:00,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:00,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:00,680 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:00,680 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:00,681 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:00,681 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:00,680 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:00,681 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:00,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:00,681 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,681 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:00,681 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:00,681 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:00,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:00,682 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:00,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54638 deadline: 1680281700679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,683 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=565 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/A/12c3b3a9efa144de90b54055674e68f5 2023-03-31 16:54:00,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:00,685 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:00,781 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:00,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:00,783 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:00,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:00,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:00,784 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:00,784 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:00,784 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:00,784 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:00,784 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:00,784 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:00,785 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:00,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54602 deadline: 1680281700779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,786 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:00,786 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:00,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54628 deadline: 1680281700780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:00,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:00,786 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:00,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54630 deadline: 1680281700780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,786 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:00,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54638 deadline: 1680281700783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,787 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:00,787 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,787 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:00,787 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:00,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:00,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:00,788 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:00,788 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:00,788 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:00,788 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:00,788 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:00,788 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:00,788 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:00,788 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:00,788 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:54:00,789 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. started... 2023-03-31 16:54:00,789 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. as already flushing 2023-03-31 16:54:00,789 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:54:00,789 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:00,789 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:00,789 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,789 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:00,789 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:00,790 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,790 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:00,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:00,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:00,791 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:00,791 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:00,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:00,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:00,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:00,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:00,793 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:00,793 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ed02853[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:00,793 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:00,793 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:00,795 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:00,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:00,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:00,796 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:00,796 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:00,796 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:00,796 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,796 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:00,797 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:00,797 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:00,797 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:00,798 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:00,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:00,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:00,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:00,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:00,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:00,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:00,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:00,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:00,800 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:00,800 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:00,801 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:00,801 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:00,801 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:00,802 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:00,802 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:00,802 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:00,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:00,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:00,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:00,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:00,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:00,803 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:00,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:00,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:00,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:00,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:00,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:00,804 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:00,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:00,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:00,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,807 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:00,807 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:00,807 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:00,807 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:00,807 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:00,807 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:00,807 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,807 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:00,808 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:00,808 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,808 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:00,808 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:00,808 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:00,808 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:00,808 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:00,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:00,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:00,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:00,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:00,907 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:00,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:00,910 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:00,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:00,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:00,912 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:00,912 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:00,912 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:00,912 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:00,912 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:00,913 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:00,921 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:00,921 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:00,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:00,921 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,921 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:00,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:00,922 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:00,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:00,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:00,922 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:00,933 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:00,933 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:00,933 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:00,933 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:00,933 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:00,933 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:00,933 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:00,933 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:54:00,933 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. started... 2023-03-31 16:54:00,933 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. as already flushing 2023-03-31 16:54:00,934 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:54:00,934 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:00,934 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:00,934 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,934 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:00,934 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:00,935 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,935 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:00,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:00,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:00,936 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:00,936 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:00,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:00,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:00,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:00,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:00,937 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:00,937 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36bfb24b[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:00,937 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:00,937 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:00,942 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:00,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:00,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:00,942 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:00,942 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:00,942 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:00,942 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,942 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:00,946 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,946 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:00,946 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:00,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:00,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:00,946 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:00,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:00,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:00,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:00,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:00,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:00,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:00,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:00,949 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:00,949 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:00,949 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:00,949 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:00,949 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:00,951 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:00,951 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:00,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:00,951 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:00,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:00,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:00,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:00,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:00,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:00,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:00,951 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:00,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:00,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:00,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:00,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:00,972 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:00,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:00,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,976 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:00,976 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:00,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:00,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:00,976 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,976 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:00,976 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:00,976 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:00,976 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:00,976 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:00,976 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,976 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:00,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:00,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:00,976 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:00,977 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:00,977 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:00,977 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:00,980 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:00,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54604 deadline: 1680281700980, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,988 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:00,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54602 deadline: 1680281700986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,989 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:00,989 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:00,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54628 deadline: 1680281700988, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54638 deadline: 1680281700988, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:00,989 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:00,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54630 deadline: 1680281700988, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:01,077 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:01,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:01,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:01,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:01,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:01,081 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:01,081 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:01,081 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:01,081 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:01,081 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:01,081 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:01,083 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:01,083 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:01,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:01,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:01,084 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:01,084 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:01,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:01,084 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:01,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:01,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:01,084 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:01,085 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:01,085 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:01,085 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:01,085 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:01,085 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:01,085 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:54:01,085 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. started... 2023-03-31 16:54:01,085 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:01,085 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. as already flushing 2023-03-31 16:54:01,085 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:54:01,085 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:01,085 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:01,085 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:01,086 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:01,086 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:01,088 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:01,088 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:01,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:01,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:01,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:01,088 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:01,088 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:01,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:01,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:01,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:01,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:01,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:01,090 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:01,090 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f2f21c5[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:01,090 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:01,090 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:01,091 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:01,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:01,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:01,091 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:01,091 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:01,091 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:01,091 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:01,091 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:01,093 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:01,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:01,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:01,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:01,093 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:01,093 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:01,093 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:01,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:01,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:01,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:01,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:01,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:01,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:01,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:01,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:01,095 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:01,095 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:01,095 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:01,095 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:01,095 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:01,095 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:01,097 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:01,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:01,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:01,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:01,097 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:01,097 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:01,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:01,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:01,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:01,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:01,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:01,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:01,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:01,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:01,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:01,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:01,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:01,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:01,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:01,102 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:01,102 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=565 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/B/6d9ed6869dc143dfb8fba14ec5a254e5 2023-03-31 16:54:01,103 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:01,107 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:01,107 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:01,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:01,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:01,107 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:01,107 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:01,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:01,107 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:01,107 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:01,108 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:01,108 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:01,108 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:01,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:01,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:01,108 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:01,108 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:01,108 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:01,108 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:01,208 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:01,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:01,210 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:01,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:01,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:01,211 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:01,211 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:01,212 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:01,212 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:01,212 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:01,212 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:01,213 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:01,213 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:01,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:01,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:01,214 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:01,214 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:01,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:01,214 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:01,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:01,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:01,214 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:01,215 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:01,215 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:01,215 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:01,215 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:01,215 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:01,215 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:01,215 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:01,215 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:54:01,215 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. started... 2023-03-31 16:54:01,215 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. as already flushing 2023-03-31 16:54:01,215 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:54:01,215 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:01,215 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:01,215 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:01,215 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:01,215 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:01,217 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:01,217 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:01,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:01,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:01,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:01,217 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:01,217 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:01,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:01,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:01,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:01,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:01,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:01,219 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:01,219 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:01,219 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7210d975[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:01,219 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:01,220 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:01,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:01,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:01,220 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:01,220 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:01,220 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:01,220 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:01,220 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:01,223 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:01,223 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:01,223 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:01,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:01,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:01,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:01,223 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:01,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:01,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:01,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:01,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:01,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:01,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:01,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:01,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:01,225 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:01,225 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:01,225 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:01,225 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:01,225 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:01,225 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:01,227 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:01,227 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:01,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:01,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:01,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:01,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:01,227 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:01,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:01,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:01,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:01,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:01,228 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:01,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:01,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:01,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:01,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:01,229 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:01,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:01,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:01,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:01,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:01,231 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:01,231 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:01,231 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:01,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:01,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:01,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:01,231 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:01,231 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:01,231 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:01,231 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:01,232 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:01,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:01,231 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:01,232 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:01,232 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:01,232 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:01,232 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:01,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:01,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:01,294 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:01,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54630 deadline: 1680281701290, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:01,294 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:01,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54628 deadline: 1680281701290, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:01,295 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:01,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54638 deadline: 1680281701292, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:01,295 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:01,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54602 deadline: 1680281701292, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:01,332 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:01,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:01,334 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:01,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:01,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:01,335 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:01,335 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:01,335 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:01,335 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:01,335 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:01,336 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:01,337 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:01,337 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:01,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:01,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:01,337 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:01,337 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:01,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:01,337 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:01,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:01,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:01,338 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:01,338 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:01,338 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:01,338 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:01,338 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:01,338 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:01,338 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:01,338 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:01,338 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:54:01,338 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. started... 2023-03-31 16:54:01,339 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. as already flushing 2023-03-31 16:54:01,339 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:54:01,339 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:01,339 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:01,339 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:01,339 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:01,339 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:01,340 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:01,340 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:01,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:01,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:01,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:01,340 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:01,340 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:01,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:01,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:01,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:01,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:01,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:01,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:01,342 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:01,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@dfdf5d4[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:01,342 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:01,343 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:01,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:01,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:01,343 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:01,343 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:01,343 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:01,343 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:01,343 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:01,346 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:01,346 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:01,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:01,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:01,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:01,346 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:01,346 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:01,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:01,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:01,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:01,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:01,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:01,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:01,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:01,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:01,349 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:01,349 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:01,349 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:01,349 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:01,349 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:01,349 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:01,350 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:01,350 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:01,351 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:01,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:01,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:01,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:01,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:01,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:01,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:01,351 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:01,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:01,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:01,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:01,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:01,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:01,352 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:01,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:01,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:01,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:01,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:01,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:01,360 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:01,360 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:01,360 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:01,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:01,360 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:01,360 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:01,360 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:01,360 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:01,360 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:01,360 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:01,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:01,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:01,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:01,360 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:01,361 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:01,361 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:01,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:01,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:01,361 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:01,461 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:01,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:01,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:01,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:01,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:01,465 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:01,466 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:01,466 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:01,466 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:01,466 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:01,466 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:01,469 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:01,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:01,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:01,469 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:01,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:01,470 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:01,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:01,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:01,471 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:01,471 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:01,471 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:01,471 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:01,471 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:01,471 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:01,472 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:01,472 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:01,472 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:01,472 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:01,472 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:54:01,472 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. started... 2023-03-31 16:54:01,472 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. as already flushing 2023-03-31 16:54:01,472 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:54:01,472 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:01,472 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:01,472 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:01,472 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:01,472 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:01,474 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:01,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:01,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:01,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:01,474 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:01,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:01,475 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:01,475 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:01,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:01,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:01,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:01,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:01,477 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:01,477 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@461b33ca[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:01,477 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:01,477 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:01,478 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:01,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:01,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:01,479 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:01,479 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:01,479 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:01,479 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:01,479 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:01,481 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:01,481 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:01,481 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:01,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:01,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:01,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:01,481 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:01,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:01,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:01,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:01,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:01,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:01,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:01,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:01,484 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:01,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54604 deadline: 1680281701482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:01,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:01,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:01,489 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:01,489 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:01,489 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:01,489 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:01,489 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:01,492 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:01,492 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:01,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:01,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:01,492 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:01,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:01,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:01,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:01,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:01,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:01,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:01,493 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:01,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:01,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:01,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:01,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:01,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:01,495 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:01,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:01,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:01,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:01,498 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:01,498 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:01,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:01,498 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:01,498 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:01,498 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:01,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:01,498 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:01,498 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:01,498 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:01,498 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:01,498 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:01,498 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:01,499 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:01,498 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:01,499 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:01,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:01,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:01,516 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=565 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/C/e16c95940b8049adb4d6fc6634a7b840 2023-03-31 16:54:01,523 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/A/12c3b3a9efa144de90b54055674e68f5 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/12c3b3a9efa144de90b54055674e68f5 2023-03-31 16:54:01,528 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/12c3b3a9efa144de90b54055674e68f5, entries=200, sequenceid=565, filesize=14.3 K 2023-03-31 16:54:01,529 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/B/6d9ed6869dc143dfb8fba14ec5a254e5 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/6d9ed6869dc143dfb8fba14ec5a254e5 2023-03-31 16:54:01,539 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/6d9ed6869dc143dfb8fba14ec5a254e5, entries=150, sequenceid=565, filesize=11.9 K 2023-03-31 16:54:01,540 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/C/e16c95940b8049adb4d6fc6634a7b840 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/e16c95940b8049adb4d6fc6634a7b840 2023-03-31 16:54:01,546 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/e16c95940b8049adb4d6fc6634a7b840, entries=150, sequenceid=565, filesize=11.9 K 2023-03-31 16:54:01,546 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~459.61 KB/470640, currentSize=26.84 KB/27480 for e94df50bb46b68756c6a704332e1c5f4 in 875ms, sequenceid=565, compaction requested=true 2023-03-31 16:54:01,547 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e94df50bb46b68756c6a704332e1c5f4: 2023-03-31 16:54:01,547 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:54:01,547 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 16:54:01,547 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 16:54:01,547 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:54:01,547 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:54:01,548 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37512 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 16:54:01,548 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): e94df50bb46b68756c6a704332e1c5f4/B is initiating minor compaction (all files) 2023-03-31 16:54:01,548 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39952 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 16:54:01,549 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1912): e94df50bb46b68756c6a704332e1c5f4/A is initiating minor compaction (all files) 2023-03-31 16:54:01,549 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e94df50bb46b68756c6a704332e1c5f4/A in TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:54:01,549 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/cf90fdeba90b46b6862e8535e4ad999a, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/8e110553cd8040b392e868d1ba25330d, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/12c3b3a9efa144de90b54055674e68f5] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp, totalSize=39.0 K 2023-03-31 16:54:01,549 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e94df50bb46b68756c6a704332e1c5f4/B in TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:54:01,549 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/a55c2c3809b14425929ae2608a09ea40, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/b2cde8036b11485d86cfbdf0994d8025, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/6d9ed6869dc143dfb8fba14ec5a254e5] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp, totalSize=36.6 K 2023-03-31 16:54:01,550 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting cf90fdeba90b46b6862e8535e4ad999a, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=526, earliestPutTs=1680281639088 2023-03-31 16:54:01,550 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting a55c2c3809b14425929ae2608a09ea40, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=526, earliestPutTs=1680281639088 2023-03-31 16:54:01,551 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 8e110553cd8040b392e868d1ba25330d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=537, earliestPutTs=1680281639203 2023-03-31 16:54:01,551 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting b2cde8036b11485d86cfbdf0994d8025, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=537, earliestPutTs=1680281639203 2023-03-31 16:54:01,551 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 6d9ed6869dc143dfb8fba14ec5a254e5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=565, earliestPutTs=1680281640356 2023-03-31 16:54:01,552 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 12c3b3a9efa144de90b54055674e68f5, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=565, earliestPutTs=1680281640356 2023-03-31 16:54:01,561 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): e94df50bb46b68756c6a704332e1c5f4#B#compaction#92 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:54:01,563 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] throttle.PressureAwareThroughputController(145): e94df50bb46b68756c6a704332e1c5f4#A#compaction#93 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:54:01,590 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/A/6623073eee49471ab9afce01e7b976ff as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/6623073eee49471ab9afce01e7b976ff 2023-03-31 16:54:01,597 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e94df50bb46b68756c6a704332e1c5f4/A of e94df50bb46b68756c6a704332e1c5f4 into 6623073eee49471ab9afce01e7b976ff(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:54:01,597 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e94df50bb46b68756c6a704332e1c5f4: 2023-03-31 16:54:01,597 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4., storeName=e94df50bb46b68756c6a704332e1c5f4/A, priority=13, startTime=1680281641547; duration=0sec 2023-03-31 16:54:01,598 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:54:01,598 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 16:54:01,599 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:01,599 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49630 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 16:54:01,599 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1912): e94df50bb46b68756c6a704332e1c5f4/C is initiating minor compaction (all files) 2023-03-31 16:54:01,599 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e94df50bb46b68756c6a704332e1c5f4/C in TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:54:01,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:01,599 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/55d8160c4f974b8cb369c3aa22d21c9b, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/bea34f5bbb6a44d28063f8259b6a826d, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/d1eaaaf307824034a64d551410fcc92f, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/e16c95940b8049adb4d6fc6634a7b840] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp, totalSize=48.5 K 2023-03-31 16:54:01,600 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 55d8160c4f974b8cb369c3aa22d21c9b, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=487, earliestPutTs=1680281638077 2023-03-31 16:54:01,600 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting bea34f5bbb6a44d28063f8259b6a826d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=526, earliestPutTs=1680281639088 2023-03-31 16:54:01,601 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting d1eaaaf307824034a64d551410fcc92f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=537, earliestPutTs=1680281639203 2023-03-31 16:54:01,601 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting e16c95940b8049adb4d6fc6634a7b840, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=565, earliestPutTs=1680281640356 2023-03-31 16:54:01,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:01,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:01,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:01,603 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:01,603 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:01,603 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:01,603 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:01,604 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:01,604 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:01,605 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:01,605 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:01,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:01,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:01,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:01,606 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:01,606 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:01,606 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:01,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:01,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:01,607 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:01,612 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:01,612 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:01,612 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:01,612 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:01,612 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:01,612 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:01,612 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:01,613 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] throttle.PressureAwareThroughputController(145): e94df50bb46b68756c6a704332e1c5f4#C#compaction#94 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:54:01,629 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:54:01,629 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. started... 2023-03-31 16:54:01,629 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e94df50bb46b68756c6a704332e1c5f4 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-03-31 16:54:01,630 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=A 2023-03-31 16:54:01,630 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:01,630 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=B 2023-03-31 16:54:01,630 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:01,630 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=C 2023-03-31 16:54:01,630 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:01,679 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/C/2dd6e6fe111a43bab26152d52701c804 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/2dd6e6fe111a43bab26152d52701c804 2023-03-31 16:54:01,691 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e94df50bb46b68756c6a704332e1c5f4/C of e94df50bb46b68756c6a704332e1c5f4 into 2dd6e6fe111a43bab26152d52701c804(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:54:01,691 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e94df50bb46b68756c6a704332e1c5f4: 2023-03-31 16:54:01,691 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4., storeName=e94df50bb46b68756c6a704332e1c5f4/C, priority=12, startTime=1680281641547; duration=0sec 2023-03-31 16:54:01,691 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:54:01,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on e94df50bb46b68756c6a704332e1c5f4 2023-03-31 16:54:01,800 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. as already flushing 2023-03-31 16:54:01,817 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680281641809, max timestamp=1680281641815Num uniques 859; 2023-03-31 16:54:01,817 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680281641809, max timestamp=1680281641815Num uniques 859; 2023-03-31 16:54:01,818 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 16:54:01,818 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 16:54:01,818 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680281641809, max timestamp=1680281641815Num uniques 859; 2023-03-31 16:54:01,818 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 16:54:01,843 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680281641830, max timestamp=1680281641840Num uniques 1068; 2023-03-31 16:54:01,843 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-03-31 16:54:01,843 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:01,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54638 deadline: 1680281701838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:01,844 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680281641830, max timestamp=1680281641840Num uniques 1068; 2023-03-31 16:54:01,844 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=6870, suffix heap size=139040,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=150 2023-03-31 16:54:01,844 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680281641830, max timestamp=1680281641840Num uniques 1068; 2023-03-31 16:54:01,844 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=6870, suffix heap size=139040,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=150 2023-03-31 16:54:01,868 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680281641861, max timestamp=1680281641865Num uniques 1027; 2023-03-31 16:54:01,868 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-03-31 16:54:01,868 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680281641861, max timestamp=1680281641865Num uniques 1009; 2023-03-31 16:54:01,868 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680281641861, max timestamp=1680281641865Num uniques 1009; 2023-03-31 16:54:01,868 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=8102, suffix heap size=132904,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=177 2023-03-31 16:54:01,868 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=8102, suffix heap size=132904,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=177 2023-03-31 16:54:01,904 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1680281641891, max timestamp=1680281641901Num uniques 1036; 2023-03-31 16:54:01,904 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=13096, suffix heap size=135712,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=286 2023-03-31 16:54:01,904 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1680281641891, max timestamp=1680281641901Num uniques 1036; 2023-03-31 16:54:01,904 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1680281641891, max timestamp=1680281641901Num uniques 1036; 2023-03-31 16:54:01,904 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=13096, suffix heap size=135712,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=286 2023-03-31 16:54:01,904 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=13096, suffix heap size=135712,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=286 2023-03-31 16:54:01,927 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:01,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54628 deadline: 1680281701923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:01,927 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:01,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54602 deadline: 1680281701925, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:01,928 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=345, cellSize=15800, totalHeapSize=36120, min timestamp=1680281641907, max timestamp=1680281641925Num uniques 1145; 2023-03-31 16:54:01,928 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=15800, suffix heap size=147048,new segment heap size=36120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=345 2023-03-31 16:54:01,930 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=345, cellSize=15800, totalHeapSize=36120, min timestamp=1680281641907, max timestamp=1680281641925Num uniques 1145; 2023-03-31 16:54:01,930 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=15800, suffix heap size=147048,new segment heap size=36120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=345 2023-03-31 16:54:01,932 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680281641907, max timestamp=1680281641925Num uniques 1145; 2023-03-31 16:54:01,933 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=6870, suffix heap size=147048,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=150 2023-03-31 16:54:01,987 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/B/7c4045ff4cb14706a0f47e99b8dba7b7 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/7c4045ff4cb14706a0f47e99b8dba7b7 2023-03-31 16:54:01,989 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:01,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54630 deadline: 1680281701988, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:01,991 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1680281641968, max timestamp=1680281641987Num uniques 1204; 2023-03-31 16:54:01,991 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=55140, new segment data size=9340, suffix heap size=153184,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1204, new segment cells count=204 2023-03-31 16:54:01,993 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1680281641968, max timestamp=1680281641987Num uniques 1204; 2023-03-31 16:54:01,993 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=55140, new segment data size=9340, suffix heap size=153184,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1204, new segment cells count=204 2023-03-31 16:54:01,994 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1680281641968, max timestamp=1680281641987Num uniques 1009; 2023-03-31 16:54:01,994 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46210, new segment data size=9340, suffix heap size=132904,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=204 2023-03-31 16:54:01,995 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e94df50bb46b68756c6a704332e1c5f4/B of e94df50bb46b68756c6a704332e1c5f4 into 7c4045ff4cb14706a0f47e99b8dba7b7(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:54:01,996 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e94df50bb46b68756c6a704332e1c5f4: 2023-03-31 16:54:01,996 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4., storeName=e94df50bb46b68756c6a704332e1c5f4/B, priority=13, startTime=1680281641547; duration=0sec 2023-03-31 16:54:01,996 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:54:02,037 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1680281642025, max timestamp=1680281642034Num uniques 1063; 2023-03-31 16:54:02,037 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=12038, suffix heap size=138520,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=263 2023-03-31 16:54:02,037 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1680281642025, max timestamp=1680281642034Num uniques 1063; 2023-03-31 16:54:02,037 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=12038, suffix heap size=138520,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=263 2023-03-31 16:54:02,037 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1680281642025, max timestamp=1680281642034Num uniques 1063; 2023-03-31 16:54:02,037 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=12038, suffix heap size=138520,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=263 2023-03-31 16:54:02,058 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:02,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54628 deadline: 1680281702054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,059 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:02,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54602 deadline: 1680281702055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,060 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1680281642052, max timestamp=1680281642057Num uniques 1122; 2023-03-31 16:54:02,060 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=51382, new segment data size=7872, suffix heap size=144656,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=172 2023-03-31 16:54:02,060 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1680281642052, max timestamp=1680281642057Num uniques 1122; 2023-03-31 16:54:02,060 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1680281642052, max timestamp=1680281642057Num uniques 1122; 2023-03-31 16:54:02,060 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=51382, new segment data size=7872, suffix heap size=144656,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=172 2023-03-31 16:54:02,060 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=51382, new segment data size=7872, suffix heap size=144656,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=172 2023-03-31 16:54:02,097 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1680281642081, max timestamp=1680281642095Num uniques 1031; 2023-03-31 16:54:02,097 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=12866, suffix heap size=135192,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=281 2023-03-31 16:54:02,098 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1680281642081, max timestamp=1680281642095Num uniques 1031; 2023-03-31 16:54:02,098 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=12866, suffix heap size=135192,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=281 2023-03-31 16:54:02,098 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1680281642081, max timestamp=1680281642095Num uniques 1031; 2023-03-31 16:54:02,098 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=12866, suffix heap size=135192,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=281 2023-03-31 16:54:02,108 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=573 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/A/3a4950b35cde4b9f997883da1dd839cb 2023-03-31 16:54:02,125 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1680281642111, max timestamp=1680281642122Num uniques 1140; 2023-03-31 16:54:02,125 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=52210, new segment data size=13280, suffix heap size=146528,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=290 2023-03-31 16:54:02,126 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1680281642111, max timestamp=1680281642122Num uniques 1140; 2023-03-31 16:54:02,126 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1680281642111, max timestamp=1680281642122Num uniques 1140; 2023-03-31 16:54:02,126 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=52210, new segment data size=13280, suffix heap size=146528,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=290 2023-03-31 16:54:02,126 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=52210, new segment data size=13280, suffix heap size=146528,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=290 2023-03-31 16:54:02,139 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=573 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/B/f736406e8e4249369153964ff57a6a2c 2023-03-31 16:54:02,154 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=573 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/C/5c2dbac42bcf4fae9fcead4456abe051 2023-03-31 16:54:02,160 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=299, cellSize=13694, totalHeapSize=31336, min timestamp=1680281642146, max timestamp=1680281642157Num uniques 1149; 2023-03-31 16:54:02,160 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=52624, new segment data size=13694, suffix heap size=147464,new segment heap size=31336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=299 2023-03-31 16:54:02,160 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=299, cellSize=13694, totalHeapSize=31336, min timestamp=1680281642146, max timestamp=1680281642157Num uniques 1149; 2023-03-31 16:54:02,160 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=299, cellSize=13694, totalHeapSize=31336, min timestamp=1680281642146, max timestamp=1680281642157Num uniques 1149; 2023-03-31 16:54:02,161 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=52624, new segment data size=13694, suffix heap size=147464,new segment heap size=31336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=299 2023-03-31 16:54:02,160 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=52624, new segment data size=13694, suffix heap size=147464,new segment heap size=31336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=299 2023-03-31 16:54:02,163 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/A/3a4950b35cde4b9f997883da1dd839cb as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/3a4950b35cde4b9f997883da1dd839cb 2023-03-31 16:54:02,186 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:02,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 313 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54638 deadline: 1680281702178, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,186 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:02,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54602 deadline: 1680281702181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,187 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:02,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54630 deadline: 1680281702184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,188 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:02,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54628 deadline: 1680281702184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,194 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/3a4950b35cde4b9f997883da1dd839cb, entries=100, sequenceid=573, filesize=9.5 K 2023-03-31 16:54:02,195 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/B/f736406e8e4249369153964ff57a6a2c as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/f736406e8e4249369153964ff57a6a2c 2023-03-31 16:54:02,199 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/f736406e8e4249369153964ff57a6a2c, entries=100, sequenceid=573, filesize=9.5 K 2023-03-31 16:54:02,200 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/C/5c2dbac42bcf4fae9fcead4456abe051 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/5c2dbac42bcf4fae9fcead4456abe051 2023-03-31 16:54:02,207 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/5c2dbac42bcf4fae9fcead4456abe051, entries=100, sequenceid=573, filesize=9.5 K 2023-03-31 16:54:02,208 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=154.31 KB/158010 for e94df50bb46b68756c6a704332e1c5f4 in 579ms, sequenceid=573, compaction requested=false 2023-03-31 16:54:02,209 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e94df50bb46b68756c6a704332e1c5f4: 2023-03-31 16:54:02,209 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:54:02,209 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:02,209 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:02,209 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,209 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:02,209 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:02,211 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,211 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:02,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:02,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:02,211 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:02,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:02,211 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:02,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:02,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:02,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:02,212 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:02,213 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@700c03c5[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:02,213 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:02,213 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:02,214 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:02,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:02,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:02,214 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:02,214 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,214 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:02,214 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:02,214 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:02,216 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:02,216 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:02,216 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,216 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:02,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:02,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:02,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:02,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:02,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:02,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:02,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:02,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:02,219 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:02,219 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:02,219 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:02,219 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:02,219 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:02,219 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:02,220 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:02,220 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:02,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:02,220 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:02,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:02,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:02,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:02,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:02,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:02,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:02,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:02,221 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:02,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:02,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:02,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:02,222 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:02,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:02,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,225 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:02,225 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,225 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:02,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:02,225 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:02,226 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:02,226 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:02,225 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:02,226 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:02,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:02,225 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:02,226 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,226 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:02,226 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:02,226 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:02,226 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:02,226 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:02,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:02,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:02,291 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e94df50bb46b68756c6a704332e1c5f4 3/3 column families, dataSize=161.02 KB heapSize=452.25 KB 2023-03-31 16:54:02,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on e94df50bb46b68756c6a704332e1c5f4 2023-03-31 16:54:02,292 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1680281642177, max timestamp=1680281642290Num uniques 1158; 2023-03-31 16:54:02,292 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=53030, new segment data size=7230, suffix heap size=148400,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1158, new segment cells count=158 2023-03-31 16:54:02,293 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1680281642177, max timestamp=1680281642290Num uniques 1158; 2023-03-31 16:54:02,293 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1680281642177, max timestamp=1680281642290Num uniques 1158; 2023-03-31 16:54:02,293 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=53030, new segment data size=7230, suffix heap size=148400,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1158, new segment cells count=158 2023-03-31 16:54:02,293 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=53030, new segment data size=7230, suffix heap size=148400,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1158, new segment cells count=158 2023-03-31 16:54:02,293 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=A 2023-03-31 16:54:02,294 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:54:02,294 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=B 2023-03-31 16:54:02,294 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:54:02,294 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=C 2023-03-31 16:54:02,294 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:54:02,309 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=786 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/A/97588e78462f4c89bd775e03002aabe9 2023-03-31 16:54:02,320 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680281642313, max timestamp=1680281642318Num uniques 859; 2023-03-31 16:54:02,320 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-31 16:54:02,321 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680281642313, max timestamp=1680281642318Num uniques 859; 2023-03-31 16:54:02,321 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-31 16:54:02,321 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680281642313, max timestamp=1680281642318Num uniques 859; 2023-03-31 16:54:02,321 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-31 16:54:02,326 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:02,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:02,329 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:02,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:02,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:02,330 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:02,330 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:02,330 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:02,330 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:02,330 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:02,330 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:02,332 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:02,332 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:02,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:02,333 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:02,333 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:02,333 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:02,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:02,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:02,333 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:02,334 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:02,334 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:02,334 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:02,334 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:02,334 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:02,334 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:02,334 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:02,334 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:54:02,334 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. started... 2023-03-31 16:54:02,334 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. as already flushing 2023-03-31 16:54:02,334 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:54:02,334 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:02,334 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:02,334 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,334 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:02,334 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:02,336 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:02,336 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:02,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:02,336 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:02,336 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:02,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:02,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:02,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:02,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:02,339 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:02,339 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@423b7c0[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:02,339 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:02,339 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:02,340 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:02,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:02,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:02,340 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:02,340 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:02,340 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,340 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:02,340 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:02,342 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,342 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:02,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 324 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54638 deadline: 1680281702339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,342 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:02,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 300 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54630 deadline: 1680281702340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,344 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7680, totalHeapSize=17712, min timestamp=1680281642337, max timestamp=1680281642341Num uniques 1118; 2023-03-31 16:54:02,344 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=51190, new segment data size=7680, suffix heap size=144240,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=168 2023-03-31 16:54:02,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:02,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:02,345 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:02,345 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:02,345 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:02,344 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7682, totalHeapSize=17712, min timestamp=1680281642337, max timestamp=1680281642341Num uniques 1118; 2023-03-31 16:54:02,344 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680281642337, max timestamp=1680281642341Num uniques 1118; 2023-03-31 16:54:02,345 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=51192, new segment data size=7682, suffix heap size=144240,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=168 2023-03-31 16:54:02,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:02,345 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=7688, suffix heap size=144240,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=168 2023-03-31 16:54:02,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:02,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:02,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:02,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:02,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:02,348 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:02,348 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:02,348 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:02,348 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:02,348 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:02,348 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:02,351 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:02,351 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:02,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:02,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:02,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:02,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:02,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:02,352 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:02,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:02,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:02,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:02,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:02,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:02,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:02,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:02,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:02,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:02,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,361 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,363 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,366 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680281642358, max timestamp=1680281642363Num uniques 1027; 2023-03-31 16:54:02,366 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-03-31 16:54:02,368 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680281642358, max timestamp=1680281642363Num uniques 1027; 2023-03-31 16:54:02,369 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-03-31 16:54:02,369 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:02,368 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680281642358, max timestamp=1680281642363Num uniques 1027; 2023-03-31 16:54:02,369 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:02,369 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-03-31 16:54:02,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:02,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:02,369 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,370 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:02,370 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:02,370 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:02,370 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:02,370 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:02,370 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:02,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:02,370 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:02,370 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:02,370 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:02,370 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:02,370 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:02,396 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1680281642382, max timestamp=1680281642393Num uniques 1086; 2023-03-31 16:54:02,396 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-03-31 16:54:02,396 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1680281642382, max timestamp=1680281642393Num uniques 1086; 2023-03-31 16:54:02,396 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1680281642382, max timestamp=1680281642393Num uniques 1086; 2023-03-31 16:54:02,396 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-03-31 16:54:02,396 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-03-31 16:54:02,426 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1680281642413, max timestamp=1680281642423Num uniques 1095; 2023-03-31 16:54:02,426 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=13510, suffix heap size=141848,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=295 2023-03-31 16:54:02,426 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1680281642413, max timestamp=1680281642423Num uniques 1095; 2023-03-31 16:54:02,426 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1680281642413, max timestamp=1680281642423Num uniques 1095; 2023-03-31 16:54:02,427 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=13510, suffix heap size=141848,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=295 2023-03-31 16:54:02,427 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=13510, suffix heap size=141848,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=295 2023-03-31 16:54:02,449 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:02,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54628 deadline: 1680281702446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,450 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:02,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 303 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54630 deadline: 1680281702447, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,450 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:02,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 327 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54638 deadline: 1680281702448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,450 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:02,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 293 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54602 deadline: 1680281702448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,470 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:02,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:02,472 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:02,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:02,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:02,473 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:02,473 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:02,473 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:02,473 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:02,474 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:02,474 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:02,475 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,475 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:02,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:02,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:02,475 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,475 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:02,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:02,476 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:02,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:02,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:02,476 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:02,477 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:02,477 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:02,477 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:02,477 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:02,477 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:02,477 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:02,477 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:02,477 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:54:02,477 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. started... 2023-03-31 16:54:02,477 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. as already flushing 2023-03-31 16:54:02,477 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:54:02,477 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:02,477 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:02,477 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,478 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:02,478 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:02,481 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,481 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:02,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:02,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:02,481 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:02,481 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:02,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:02,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:02,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:02,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:02,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:02,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6692b1c5[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:02,483 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:02,483 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:02,484 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:02,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:02,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:02,484 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:02,484 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:02,484 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:02,484 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,485 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:02,486 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:02,486 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:02,486 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,486 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:02,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:02,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:02,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:02,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:02,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:02,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:02,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:02,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:02,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:02,489 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:02,489 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:02,489 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:02,489 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:02,489 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:02,490 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:02,491 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:02,491 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:02,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:02,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:02,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:02,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:02,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:02,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:02,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:02,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:02,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:02,492 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:02,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:02,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:02,493 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:02,493 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:02,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54604 deadline: 1680281702490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:02,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,496 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,496 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:02,496 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:02,496 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,496 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:02,496 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:02,496 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:02,496 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:02,496 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:02,496 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:02,496 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:02,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:02,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:02,496 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:02,497 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:02,496 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:02,497 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:02,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:02,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:02,553 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:02,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54628 deadline: 1680281702551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,553 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:02,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 305 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54630 deadline: 1680281702551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,554 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:02,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 329 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54638 deadline: 1680281702551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,554 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:02,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 295 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54602 deadline: 1680281702551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,597 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:02,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:02,599 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:02,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:02,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:02,600 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:02,600 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:02,601 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:02,601 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:02,601 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:02,601 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:02,602 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:02,602 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:02,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:02,603 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,603 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:02,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:02,603 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:02,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:02,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:02,604 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:02,604 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:02,604 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:02,604 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:02,604 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:02,604 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:02,604 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:02,604 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:02,604 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:54:02,604 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. started... 2023-03-31 16:54:02,605 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. as already flushing 2023-03-31 16:54:02,605 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:54:02,605 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:02,605 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:02,605 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,605 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:02,605 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:02,606 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:02,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:02,606 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:02,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:02,607 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:02,607 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:02,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:02,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:02,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:02,608 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:02,608 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79084bbe[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:02,608 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:02,608 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:02,610 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:02,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:02,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:02,610 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:02,610 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:02,610 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:02,610 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,610 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:02,612 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,612 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:02,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:02,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:02,612 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:02,613 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:02,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:02,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:02,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:02,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:02,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:02,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:02,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:02,615 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:02,615 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:02,615 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:02,615 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:02,615 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:02,616 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:02,616 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:02,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:02,617 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:02,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:02,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:02,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:02,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:02,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:02,617 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:02,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:02,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:02,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:02,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:02,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:02,618 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:02,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:02,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,620 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,620 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:02,620 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:02,620 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:02,620 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:02,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:02,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:02,620 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,621 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:02,621 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:02,621 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:02,621 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:02,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:02,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:02,621 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:02,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:02,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:02,621 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:02,621 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:02,721 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:02,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:02,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:02,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:02,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:02,725 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:02,725 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:02,725 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:02,725 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:02,725 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:02,725 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:02,727 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,727 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:02,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:02,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:02,727 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,727 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:02,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:02,728 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:02,728 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=786 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/B/893c33bb28f7464289213f4b6ca1bbb7 2023-03-31 16:54:02,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:02,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:02,728 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:02,728 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:02,728 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:02,729 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:02,729 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:02,729 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:02,729 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:02,729 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:02,729 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:54:02,729 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. started... 2023-03-31 16:54:02,729 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. as already flushing 2023-03-31 16:54:02,729 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:54:02,729 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:02,729 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:02,729 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,729 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:02,729 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:02,733 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:02,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:02,733 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:02,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:02,734 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:02,734 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:02,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:02,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:02,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:02,736 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:02,736 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@698a6d71[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:02,736 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:02,736 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:02,738 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:02,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:02,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:02,738 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:02,738 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:02,738 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:02,739 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:02,739 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:02,740 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:02,740 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,741 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:02,741 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:02,741 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:02,741 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:02,741 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:02,741 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:02,741 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:02,743 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:02,743 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:02,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:02,743 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:02,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:02,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:02,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:02,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:02,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:02,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:02,746 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:02,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:02,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:02,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:02,748 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=786 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/C/e3776ff91f63473dab8744134928d105 2023-03-31 16:54:02,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:02,749 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:02,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:02,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,753 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:02,753 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:02,753 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,753 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:02,753 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:02,753 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,753 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:02,753 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:02,753 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:02,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:02,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:02,753 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:02,753 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:02,753 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:02,753 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:02,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:02,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:02,753 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:02,754 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:02,754 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/A/97588e78462f4c89bd775e03002aabe9 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/97588e78462f4c89bd775e03002aabe9 2023-03-31 16:54:02,755 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:02,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54628 deadline: 1680281702754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,757 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:02,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 307 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54630 deadline: 1680281702756, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,757 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:02,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 297 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54602 deadline: 1680281702756, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,758 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:02,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 331 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54638 deadline: 1680281702757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,760 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/97588e78462f4c89bd775e03002aabe9, entries=150, sequenceid=786, filesize=11.9 K 2023-03-31 16:54:02,761 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/B/893c33bb28f7464289213f4b6ca1bbb7 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/893c33bb28f7464289213f4b6ca1bbb7 2023-03-31 16:54:02,767 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/893c33bb28f7464289213f4b6ca1bbb7, entries=150, sequenceid=786, filesize=11.9 K 2023-03-31 16:54:02,768 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/C/e3776ff91f63473dab8744134928d105 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/e3776ff91f63473dab8744134928d105 2023-03-31 16:54:02,774 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/e3776ff91f63473dab8744134928d105, entries=150, sequenceid=786, filesize=11.9 K 2023-03-31 16:54:02,775 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~86.20 KB/88272, currentSize=154.31 KB/158010 for e94df50bb46b68756c6a704332e1c5f4 in 484ms, sequenceid=786, compaction requested=true 2023-03-31 16:54:02,776 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e94df50bb46b68756c6a704332e1c5f4: 2023-03-31 16:54:02,776 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:54:02,776 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 16:54:02,776 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 16:54:02,776 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:54:02,776 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:54:02,779 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35170 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 16:54:02,779 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1912): e94df50bb46b68756c6a704332e1c5f4/A is initiating minor compaction (all files) 2023-03-31 16:54:02,779 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e94df50bb46b68756c6a704332e1c5f4/A in TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:54:02,779 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/6623073eee49471ab9afce01e7b976ff, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/3a4950b35cde4b9f997883da1dd839cb, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/97588e78462f4c89bd775e03002aabe9] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp, totalSize=34.3 K 2023-03-31 16:54:02,779 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35170 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 16:54:02,779 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): e94df50bb46b68756c6a704332e1c5f4/B is initiating minor compaction (all files) 2023-03-31 16:54:02,779 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e94df50bb46b68756c6a704332e1c5f4/B in TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:54:02,779 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/7c4045ff4cb14706a0f47e99b8dba7b7, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/f736406e8e4249369153964ff57a6a2c, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/893c33bb28f7464289213f4b6ca1bbb7] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp, totalSize=34.3 K 2023-03-31 16:54:02,780 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 6623073eee49471ab9afce01e7b976ff, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=565, earliestPutTs=1680281640356 2023-03-31 16:54:02,780 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 7c4045ff4cb14706a0f47e99b8dba7b7, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=565, earliestPutTs=1680281640356 2023-03-31 16:54:02,780 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 3a4950b35cde4b9f997883da1dd839cb, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=573, earliestPutTs=1680281640675 2023-03-31 16:54:02,781 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting f736406e8e4249369153964ff57a6a2c, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=573, earliestPutTs=1680281640675 2023-03-31 16:54:02,781 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 97588e78462f4c89bd775e03002aabe9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=786, earliestPutTs=1680281642177 2023-03-31 16:54:02,781 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 893c33bb28f7464289213f4b6ca1bbb7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=786, earliestPutTs=1680281642177 2023-03-31 16:54:02,789 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] throttle.PressureAwareThroughputController(145): e94df50bb46b68756c6a704332e1c5f4#A#compaction#101 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:54:02,791 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): e94df50bb46b68756c6a704332e1c5f4#B#compaction#102 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:54:02,854 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:02,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:02,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:02,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:02,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:02,857 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:02,857 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:02,857 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:02,857 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:02,858 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:02,858 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:02,861 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,861 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:02,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:02,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:02,862 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:02,862 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:02,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:02,862 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:02,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:02,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:02,863 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:02,863 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:02,863 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:02,863 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:02,863 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:02,863 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:02,864 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:02,864 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:02,864 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:54:02,864 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. started... 2023-03-31 16:54:02,864 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e94df50bb46b68756c6a704332e1c5f4 3/3 column families, dataSize=154.31 KB heapSize=432 KB 2023-03-31 16:54:02,864 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=A 2023-03-31 16:54:02,864 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:54:02,865 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=B 2023-03-31 16:54:02,865 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:54:02,865 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=C 2023-03-31 16:54:02,865 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:54:02,875 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=892 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/A/2a5bdc8ee19a4f81980960dcc74363d6 2023-03-31 16:54:03,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on e94df50bb46b68756c6a704332e1c5f4 2023-03-31 16:54:03,061 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. as already flushing 2023-03-31 16:54:03,063 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:03,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54628 deadline: 1680281703061, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:03,064 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:03,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 300 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54602 deadline: 1680281703062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:03,065 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:03,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 334 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54638 deadline: 1680281703062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:03,065 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:03,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 310 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54630 deadline: 1680281703062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:03,167 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:03,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54628 deadline: 1680281703164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:03,167 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:03,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 302 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54602 deadline: 1680281703166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:03,170 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:03,170 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:03,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 336 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54638 deadline: 1680281703166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:03,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 312 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54630 deadline: 1680281703166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:03,201 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/A/5f59c5cfe19b4b8bae02f8dee8732980 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/5f59c5cfe19b4b8bae02f8dee8732980 2023-03-31 16:54:03,211 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/B/750df53f91394e0f86227e491812442f as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/750df53f91394e0f86227e491812442f 2023-03-31 16:54:03,214 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e94df50bb46b68756c6a704332e1c5f4/A of e94df50bb46b68756c6a704332e1c5f4 into 5f59c5cfe19b4b8bae02f8dee8732980(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:54:03,214 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e94df50bb46b68756c6a704332e1c5f4: 2023-03-31 16:54:03,214 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4., storeName=e94df50bb46b68756c6a704332e1c5f4/A, priority=13, startTime=1680281642776; duration=0sec 2023-03-31 16:54:03,214 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:54:03,214 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 16:54:03,216 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35136 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 16:54:03,216 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1912): e94df50bb46b68756c6a704332e1c5f4/C is initiating minor compaction (all files) 2023-03-31 16:54:03,216 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e94df50bb46b68756c6a704332e1c5f4/C in TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:54:03,217 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/2dd6e6fe111a43bab26152d52701c804, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/5c2dbac42bcf4fae9fcead4456abe051, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/e3776ff91f63473dab8744134928d105] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp, totalSize=34.3 K 2023-03-31 16:54:03,217 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 2dd6e6fe111a43bab26152d52701c804, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=565, earliestPutTs=1680281640356 2023-03-31 16:54:03,218 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 5c2dbac42bcf4fae9fcead4456abe051, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=573, earliestPutTs=1680281640675 2023-03-31 16:54:03,218 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting e3776ff91f63473dab8744134928d105, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=786, earliestPutTs=1680281642177 2023-03-31 16:54:03,219 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e94df50bb46b68756c6a704332e1c5f4/B of e94df50bb46b68756c6a704332e1c5f4 into 750df53f91394e0f86227e491812442f(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:54:03,219 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e94df50bb46b68756c6a704332e1c5f4: 2023-03-31 16:54:03,219 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4., storeName=e94df50bb46b68756c6a704332e1c5f4/B, priority=13, startTime=1680281642776; duration=0sec 2023-03-31 16:54:03,219 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:54:03,227 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] throttle.PressureAwareThroughputController(145): e94df50bb46b68756c6a704332e1c5f4#C#compaction#105 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:54:03,295 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=892 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/B/132aca2b84f94772b78de924229fa2c9 2023-03-31 16:54:03,369 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:03,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54628 deadline: 1680281703368, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:03,371 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:03,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 304 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54602 deadline: 1680281703369, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:03,372 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:03,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 314 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54630 deadline: 1680281703371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:03,373 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:03,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 338 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54638 deadline: 1680281703372, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:03,638 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/C/5a42f3222f8d460687b1f6da4283918f as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/5a42f3222f8d460687b1f6da4283918f 2023-03-31 16:54:03,644 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e94df50bb46b68756c6a704332e1c5f4/C of e94df50bb46b68756c6a704332e1c5f4 into 5a42f3222f8d460687b1f6da4283918f(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:54:03,644 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e94df50bb46b68756c6a704332e1c5f4: 2023-03-31 16:54:03,644 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4., storeName=e94df50bb46b68756c6a704332e1c5f4/C, priority=13, startTime=1680281642776; duration=0sec 2023-03-31 16:54:03,644 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:54:03,673 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:03,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54628 deadline: 1680281703672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:03,676 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:03,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 306 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54602 deadline: 1680281703674, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:03,678 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:03,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 316 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54630 deadline: 1680281703675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:03,678 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:03,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 340 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54638 deadline: 1680281703675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:03,707 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=892 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/C/66ba1c9fbf04488eabf00d167833ae72 2023-03-31 16:54:03,712 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/A/2a5bdc8ee19a4f81980960dcc74363d6 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/2a5bdc8ee19a4f81980960dcc74363d6 2023-03-31 16:54:03,717 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/2a5bdc8ee19a4f81980960dcc74363d6, entries=150, sequenceid=892, filesize=11.9 K 2023-03-31 16:54:03,717 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/B/132aca2b84f94772b78de924229fa2c9 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/132aca2b84f94772b78de924229fa2c9 2023-03-31 16:54:03,722 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/132aca2b84f94772b78de924229fa2c9, entries=150, sequenceid=892, filesize=11.9 K 2023-03-31 16:54:03,723 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/C/66ba1c9fbf04488eabf00d167833ae72 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/66ba1c9fbf04488eabf00d167833ae72 2023-03-31 16:54:03,728 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/66ba1c9fbf04488eabf00d167833ae72, entries=150, sequenceid=892, filesize=11.9 K 2023-03-31 16:54:03,728 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~431.95 KB/442320, currentSize=26.84 KB/27480 for e94df50bb46b68756c6a704332e1c5f4 in 864ms, sequenceid=892, compaction requested=false 2023-03-31 16:54:03,729 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e94df50bb46b68756c6a704332e1c5f4: 2023-03-31 16:54:03,729 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:54:03,729 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:03,729 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:03,729 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:03,729 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:03,729 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:03,730 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:03,730 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:03,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:03,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:03,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:03,731 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:03,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:03,731 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:03,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:03,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:03,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:03,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:03,733 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:03,733 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@700e4a37[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:03,733 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:03,733 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:03,735 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:03,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:03,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:03,736 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:03,736 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:03,736 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:03,736 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:03,736 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:03,737 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:03,737 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:03,737 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:03,737 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:03,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:03,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:03,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:03,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:03,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:03,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:03,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:03,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:03,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:03,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:03,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:03,740 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:03,740 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:03,740 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:03,740 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:03,740 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:03,740 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:03,741 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:03,741 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:03,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:03,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:03,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:03,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:03,741 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:03,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:03,742 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:03,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:03,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:03,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:03,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:03,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:03,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:03,743 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:03,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:03,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:03,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:03,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:03,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:03,746 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:03,746 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:03,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:03,746 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:03,746 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:03,746 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:03,746 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:03,746 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:03,746 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:03,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:03,746 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:03,746 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:03,746 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:03,746 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:03,746 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:03,746 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:03,746 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:03,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:03,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:03,847 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:03,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:03,849 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:03,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:03,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:03,850 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:03,850 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:03,850 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:03,850 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:03,851 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:03,851 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:03,853 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:03,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:03,853 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:03,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:03,853 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:03,853 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:03,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:03,854 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:03,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:03,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:03,854 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:03,855 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:03,855 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:03,855 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:03,855 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:03,855 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:03,855 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:03,855 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:03,855 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:54:03,855 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. started... 2023-03-31 16:54:03,855 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e94df50bb46b68756c6a704332e1c5f4 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-03-31 16:54:03,856 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=A 2023-03-31 16:54:03,856 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:03,856 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=B 2023-03-31 16:54:03,856 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:03,856 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=C 2023-03-31 16:54:03,856 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:04,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on e94df50bb46b68756c6a704332e1c5f4 2023-03-31 16:54:04,180 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. as already flushing 2023-03-31 16:54:04,194 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680281644189, max timestamp=1680281644192Num uniques 859; 2023-03-31 16:54:04,194 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680281644189, max timestamp=1680281644192Num uniques 859; 2023-03-31 16:54:04,194 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680281644189, max timestamp=1680281644192Num uniques 859; 2023-03-31 16:54:04,194 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 16:54:04,194 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 16:54:04,195 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 16:54:04,208 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=418, cellSize=19138, totalHeapSize=43712, min timestamp=1680281644196, max timestamp=1680281644206Num uniques 1018; 2023-03-31 16:54:04,208 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=19138, suffix heap size=133840,new segment heap size=43712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=418 2023-03-31 16:54:04,208 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=418, cellSize=19138, totalHeapSize=43712, min timestamp=1680281644196, max timestamp=1680281644206Num uniques 1018; 2023-03-31 16:54:04,208 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=418, cellSize=19138, totalHeapSize=43712, min timestamp=1680281644196, max timestamp=1680281644206Num uniques 1018; 2023-03-31 16:54:04,208 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=19138, suffix heap size=133840,new segment heap size=43712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=418 2023-03-31 16:54:04,208 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=19138, suffix heap size=133840,new segment heap size=43712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=418 2023-03-31 16:54:04,224 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:04,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 319 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54602 deadline: 1680281704221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:04,225 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:04,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54628 deadline: 1680281704221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:04,225 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:04,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 331 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54630 deadline: 1680281704222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:04,226 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:04,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 354 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54638 deadline: 1680281704224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:04,264 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=902 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/A/9974ca78841945c79636d441f17a71ac 2023-03-31 16:54:04,280 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=902 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/B/8afb7f28e8c2411bb5a998d593ae557a 2023-03-31 16:54:04,327 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:04,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 321 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54602 deadline: 1680281704325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:04,327 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:04,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 292 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54628 deadline: 1680281704326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:04,328 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:04,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 333 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54630 deadline: 1680281704326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:04,328 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:04,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 356 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54638 deadline: 1680281704327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:04,507 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:04,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54604 deadline: 1680281704506, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:04,509 DEBUG [Thread-147] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4147 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4., hostname=jenkins-hbase4.apache.org,33441,1680281619375, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-31 16:54:04,529 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:04,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 323 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54602 deadline: 1680281704528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:04,530 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:04,530 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:04,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 294 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54628 deadline: 1680281704528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:04,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 335 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54630 deadline: 1680281704529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:04,530 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:04,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 358 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54638 deadline: 1680281704529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:04,550 DEBUG [Thread-154] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x04fb2b82 to 127.0.0.1:51916 2023-03-31 16:54:04,550 DEBUG [Thread-154] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 16:54:04,551 DEBUG [Thread-152] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1818ad0a to 127.0.0.1:51916 2023-03-31 16:54:04,551 DEBUG [Thread-152] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 16:54:04,556 DEBUG [Thread-158] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4b02eeb3 to 127.0.0.1:51916 2023-03-31 16:54:04,556 DEBUG [Thread-158] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 16:54:04,558 DEBUG [Thread-156] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6129209e to 127.0.0.1:51916 2023-03-31 16:54:04,558 DEBUG [Thread-156] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 16:54:04,692 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=902 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/C/82a8878f3ac943c3806450ed60be9e5c 2023-03-31 16:54:04,698 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/A/9974ca78841945c79636d441f17a71ac as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/9974ca78841945c79636d441f17a71ac 2023-03-31 16:54:04,703 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/9974ca78841945c79636d441f17a71ac, entries=150, sequenceid=902, filesize=11.9 K 2023-03-31 16:54:04,703 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/B/8afb7f28e8c2411bb5a998d593ae557a as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/8afb7f28e8c2411bb5a998d593ae557a 2023-03-31 16:54:04,708 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/8afb7f28e8c2411bb5a998d593ae557a, entries=150, sequenceid=902, filesize=11.9 K 2023-03-31 16:54:04,709 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/C/82a8878f3ac943c3806450ed60be9e5c as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/82a8878f3ac943c3806450ed60be9e5c 2023-03-31 16:54:04,715 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/82a8878f3ac943c3806450ed60be9e5c, entries=150, sequenceid=902, filesize=11.9 K 2023-03-31 16:54:04,716 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=161.02 KB/164880 for e94df50bb46b68756c6a704332e1c5f4 in 861ms, sequenceid=902, compaction requested=true 2023-03-31 16:54:04,716 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e94df50bb46b68756c6a704332e1c5f4: 2023-03-31 16:54:04,716 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:54:04,716 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:04,716 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:04,716 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:04,716 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:04,716 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:04,718 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:04,718 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:04,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:04,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:04,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:04,718 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:04,718 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:04,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:04,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:04,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:04,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:04,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:04,719 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:04,719 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@361c16a[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:04,719 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:04,719 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:04,721 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:04,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:04,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:04,722 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:04,722 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:04,722 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:04,722 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:04,722 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:04,723 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:04,723 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:04,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:04,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:04,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:04,723 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:04,723 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:04,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:04,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:04,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:04,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:04,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:04,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:04,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:04,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:04,725 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:04,725 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:04,726 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:04,726 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:04,726 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:04,726 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:04,727 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:04,727 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:04,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:04,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:04,727 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:04,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:04,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:04,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:04,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:04,728 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:04,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:04,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:04,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:04,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:04,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:04,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:04,729 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:04,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:04,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:04,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:04,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:04,731 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:04,731 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:04,731 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:04,731 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:04,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:04,731 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:04,731 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:04,731 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:04,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:04,731 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:04,731 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:04,731 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:04,731 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:04,731 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:04,731 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:04,732 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:04,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:04,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:04,832 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:04,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on e94df50bb46b68756c6a704332e1c5f4 2023-03-31 16:54:04,832 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e94df50bb46b68756c6a704332e1c5f4 3/3 column families, dataSize=167.72 KB heapSize=460.31 KB 2023-03-31 16:54:04,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:04,833 DEBUG [Thread-149] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x09a575b8 to 127.0.0.1:51916 2023-03-31 16:54:04,833 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=A 2023-03-31 16:54:04,833 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:54:04,833 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=B 2023-03-31 16:54:04,833 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:54:04,833 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=C 2023-03-31 16:54:04,833 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:54:04,833 DEBUG [Thread-149] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 16:54:04,834 DEBUG [Thread-143] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0e16d41b to 127.0.0.1:51916 2023-03-31 16:54:04,834 DEBUG [Thread-145] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x77648431 to 127.0.0.1:51916 2023-03-31 16:54:04,834 DEBUG [Thread-143] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 16:54:04,835 DEBUG [Thread-145] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 16:54:04,835 DEBUG [Thread-141] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1b886c8f to 127.0.0.1:51916 2023-03-31 16:54:04,835 DEBUG [Thread-141] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 16:54:05,249 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=55.91 KB at sequenceid=956 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/A/13a724cb76c349a49dc8c74c8dc059f1 2023-03-31 16:54:05,666 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=55.91 KB at sequenceid=956 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/B/a0db6a3e2a554e16aedeedc0d35b2283 2023-03-31 16:54:06,077 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=55.91 KB at sequenceid=956 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/C/a789411c10b5444ea2ee03e40e33e17a 2023-03-31 16:54:06,082 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/A/13a724cb76c349a49dc8c74c8dc059f1 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/13a724cb76c349a49dc8c74c8dc059f1 2023-03-31 16:54:06,086 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/13a724cb76c349a49dc8c74c8dc059f1, entries=150, sequenceid=956, filesize=11.9 K 2023-03-31 16:54:06,086 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/B/a0db6a3e2a554e16aedeedc0d35b2283 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/a0db6a3e2a554e16aedeedc0d35b2283 2023-03-31 16:54:06,091 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/a0db6a3e2a554e16aedeedc0d35b2283, entries=150, sequenceid=956, filesize=11.9 K 2023-03-31 16:54:06,091 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/C/a789411c10b5444ea2ee03e40e33e17a as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/a789411c10b5444ea2ee03e40e33e17a 2023-03-31 16:54:06,095 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/a789411c10b5444ea2ee03e40e33e17a, entries=150, sequenceid=956, filesize=11.9 K 2023-03-31 16:54:06,096 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~167.72 KB/171750, heapSize ~460.27 KB/471312, currentSize=20.13 KB/20610 for e94df50bb46b68756c6a704332e1c5f4 in 1264ms, sequenceid=956, compaction requested=true 2023-03-31 16:54:06,096 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e94df50bb46b68756c6a704332e1c5f4: 2023-03-31 16:54:06,096 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:54:06,096 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 16:54:06,096 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:54:06,096 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 16:54:06,096 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:54:06,098 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49902 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 16:54:06,098 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1912): e94df50bb46b68756c6a704332e1c5f4/A is initiating minor compaction (all files) 2023-03-31 16:54:06,098 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49902 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 16:54:06,098 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e94df50bb46b68756c6a704332e1c5f4/A in TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:54:06,098 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): e94df50bb46b68756c6a704332e1c5f4/B is initiating minor compaction (all files) 2023-03-31 16:54:06,098 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e94df50bb46b68756c6a704332e1c5f4/B in TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:54:06,098 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/5f59c5cfe19b4b8bae02f8dee8732980, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/2a5bdc8ee19a4f81980960dcc74363d6, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/9974ca78841945c79636d441f17a71ac, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/13a724cb76c349a49dc8c74c8dc059f1] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp, totalSize=48.7 K 2023-03-31 16:54:06,098 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/750df53f91394e0f86227e491812442f, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/132aca2b84f94772b78de924229fa2c9, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/8afb7f28e8c2411bb5a998d593ae557a, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/a0db6a3e2a554e16aedeedc0d35b2283] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp, totalSize=48.7 K 2023-03-31 16:54:06,098 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 5f59c5cfe19b4b8bae02f8dee8732980, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=786, earliestPutTs=1680281642177 2023-03-31 16:54:06,099 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 750df53f91394e0f86227e491812442f, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=786, earliestPutTs=1680281642177 2023-03-31 16:54:06,099 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 2a5bdc8ee19a4f81980960dcc74363d6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=892, earliestPutTs=1680281642437 2023-03-31 16:54:06,099 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 132aca2b84f94772b78de924229fa2c9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=892, earliestPutTs=1680281642437 2023-03-31 16:54:06,099 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 9974ca78841945c79636d441f17a71ac, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=902, earliestPutTs=1680281643060 2023-03-31 16:54:06,099 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 8afb7f28e8c2411bb5a998d593ae557a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=902, earliestPutTs=1680281643060 2023-03-31 16:54:06,100 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 13a724cb76c349a49dc8c74c8dc059f1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=956, earliestPutTs=1680281644216 2023-03-31 16:54:06,100 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting a0db6a3e2a554e16aedeedc0d35b2283, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=956, earliestPutTs=1680281644216 2023-03-31 16:54:06,109 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] throttle.PressureAwareThroughputController(145): e94df50bb46b68756c6a704332e1c5f4#A#compaction#113 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:54:06,110 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): e94df50bb46b68756c6a704332e1c5f4#B#compaction#114 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:54:06,519 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/A/d881ce8497e044ed882a99d2520ca34b as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/d881ce8497e044ed882a99d2520ca34b 2023-03-31 16:54:06,522 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/B/b46068808d414eeb9f901cb44cf490b0 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/b46068808d414eeb9f901cb44cf490b0 2023-03-31 16:54:06,525 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e94df50bb46b68756c6a704332e1c5f4/A of e94df50bb46b68756c6a704332e1c5f4 into d881ce8497e044ed882a99d2520ca34b(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:54:06,525 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e94df50bb46b68756c6a704332e1c5f4: 2023-03-31 16:54:06,525 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4., storeName=e94df50bb46b68756c6a704332e1c5f4/A, priority=12, startTime=1680281646096; duration=0sec 2023-03-31 16:54:06,526 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:54:06,526 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 16:54:06,527 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49868 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 16:54:06,527 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1912): e94df50bb46b68756c6a704332e1c5f4/C is initiating minor compaction (all files) 2023-03-31 16:54:06,527 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e94df50bb46b68756c6a704332e1c5f4/C in TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:54:06,527 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/5a42f3222f8d460687b1f6da4283918f, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/66ba1c9fbf04488eabf00d167833ae72, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/82a8878f3ac943c3806450ed60be9e5c, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/a789411c10b5444ea2ee03e40e33e17a] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp, totalSize=48.7 K 2023-03-31 16:54:06,528 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 5a42f3222f8d460687b1f6da4283918f, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=786, earliestPutTs=1680281642177 2023-03-31 16:54:06,528 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e94df50bb46b68756c6a704332e1c5f4/B of e94df50bb46b68756c6a704332e1c5f4 into b46068808d414eeb9f901cb44cf490b0(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:54:06,528 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e94df50bb46b68756c6a704332e1c5f4: 2023-03-31 16:54:06,528 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4., storeName=e94df50bb46b68756c6a704332e1c5f4/B, priority=12, startTime=1680281646096; duration=0sec 2023-03-31 16:54:06,528 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 66ba1c9fbf04488eabf00d167833ae72, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=892, earliestPutTs=1680281642437 2023-03-31 16:54:06,528 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:54:06,529 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 82a8878f3ac943c3806450ed60be9e5c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=902, earliestPutTs=1680281643060 2023-03-31 16:54:06,529 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting a789411c10b5444ea2ee03e40e33e17a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=956, earliestPutTs=1680281644216 2023-03-31 16:54:06,538 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] throttle.PressureAwareThroughputController(145): e94df50bb46b68756c6a704332e1c5f4#C#compaction#115 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:54:06,547 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/C/e3d1fcf2e082486988df6496a1bd0813 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/e3d1fcf2e082486988df6496a1bd0813 2023-03-31 16:54:06,552 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e94df50bb46b68756c6a704332e1c5f4/C of e94df50bb46b68756c6a704332e1c5f4 into e3d1fcf2e082486988df6496a1bd0813(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:54:06,552 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e94df50bb46b68756c6a704332e1c5f4: 2023-03-31 16:54:06,552 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4., storeName=e94df50bb46b68756c6a704332e1c5f4/C, priority=12, startTime=1680281646096; duration=0sec 2023-03-31 16:54:06,552 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:54:08,526 DEBUG [Thread-147] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7e728705 to 127.0.0.1:51916 2023-03-31 16:54:08,527 DEBUG [Thread-147] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 16:54:08,527 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-31 16:54:08,527 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(394): wrote 187 2023-03-31 16:54:08,527 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(394): wrote 164 2023-03-31 16:54:08,527 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(394): wrote 222 2023-03-31 16:54:08,527 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(394): wrote 81 2023-03-31 16:54:08,527 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(394): wrote 191 2023-03-31 16:54:08,527 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-31 16:54:08,527 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(398): read 3941 2023-03-31 16:54:08,527 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(398): read 3988 2023-03-31 16:54:08,527 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-31 16:54:08,527 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(402): scanned 1720 2023-03-31 16:54:08,527 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(403): verified 5153 rows 2023-03-31 16:54:08,527 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(402): scanned 1733 2023-03-31 16:54:08,527 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(403): verified 5196 rows 2023-03-31 16:54:08,528 INFO [Listener at localhost/44135] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-31 16:54:08,528 DEBUG [Listener at localhost/44135] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2678d495 to 127.0.0.1:51916 2023-03-31 16:54:08,528 DEBUG [Listener at localhost/44135] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 16:54:08,532 INFO [Listener at localhost/44135] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-31 16:54:08,537 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-03-31 16:54:08,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-31 16:54:08,546 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680281648546"}]},"ts":"1680281648546"} 2023-03-31 16:54:08,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-31 16:54:08,547 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-31 16:54:08,549 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-31 16:54:08,550 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e94df50bb46b68756c6a704332e1c5f4, UNASSIGN}] 2023-03-31 16:54:08,551 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e94df50bb46b68756c6a704332e1c5f4, UNASSIGN 2023-03-31 16:54:08,552 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=e94df50bb46b68756c6a704332e1c5f4, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:08,552 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680281648552"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680281648552"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680281648552"}]},"ts":"1680281648552"} 2023-03-31 16:54:08,554 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; CloseRegionProcedure e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375}] 2023-03-31 16:54:08,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-31 16:54:08,711 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close e94df50bb46b68756c6a704332e1c5f4 2023-03-31 16:54:08,711 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing e94df50bb46b68756c6a704332e1c5f4, disabling compactions & flushes 2023-03-31 16:54:08,711 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:54:08,711 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:54:08,711 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. after waiting 0 ms 2023-03-31 16:54:08,711 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:54:08,711 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing e94df50bb46b68756c6a704332e1c5f4 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-03-31 16:54:08,712 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=A 2023-03-31 16:54:08,712 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:08,712 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=B 2023-03-31 16:54:08,712 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:08,712 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e94df50bb46b68756c6a704332e1c5f4, store=C 2023-03-31 16:54:08,712 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:08,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-31 16:54:09,119 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=966 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/A/c61f28c2e95c4dde9a0cb46609ef5208 2023-03-31 16:54:09,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-31 16:54:09,530 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=966 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/B/59b0f8c6e2d94198a9be44d67304f04c 2023-03-31 16:54:09,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-31 16:54:09,940 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=966 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/C/5a96250256424f1bac22e9e1925e8d98 2023-03-31 16:54:09,945 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/A/c61f28c2e95c4dde9a0cb46609ef5208 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/c61f28c2e95c4dde9a0cb46609ef5208 2023-03-31 16:54:09,949 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/c61f28c2e95c4dde9a0cb46609ef5208, entries=100, sequenceid=966, filesize=9.5 K 2023-03-31 16:54:09,950 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/B/59b0f8c6e2d94198a9be44d67304f04c as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/59b0f8c6e2d94198a9be44d67304f04c 2023-03-31 16:54:09,954 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/59b0f8c6e2d94198a9be44d67304f04c, entries=100, sequenceid=966, filesize=9.5 K 2023-03-31 16:54:09,954 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/.tmp/C/5a96250256424f1bac22e9e1925e8d98 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/5a96250256424f1bac22e9e1925e8d98 2023-03-31 16:54:09,958 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/5a96250256424f1bac22e9e1925e8d98, entries=100, sequenceid=966, filesize=9.5 K 2023-03-31 16:54:09,959 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for e94df50bb46b68756c6a704332e1c5f4 in 1248ms, sequenceid=966, compaction requested=false 2023-03-31 16:54:09,979 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/010cb846e12645c2ab85d0cc9628bbd3, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/a19e77acb6394df3a197abed58be738e, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/46b76279f65a48fc99c1179eab1a7971, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/ec500428d0c4460dab6c901550e705e4, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/fa6db4443ac7493a98563caf2a8e7b40, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/15cfa5e5c30442e2a7602aa3f38a1295, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/c447ed0d8629424e93db831f3698ec0e, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/07320ee76d544ccea3a912d574b89806, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/68827ed2391e4ca29a272cb933a5cedd, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/ca36732fc7d04a5caee63a13f295f814, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/d61f5d3191e243a481f2b953ec45220a, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/81256392dfb144279664831b3777c42e, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/81fd752e0ce5425ba2e00f25e99baeb0, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/dba0bdcef83a46b1abacb7a889a8f992, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/f802b521ea6349c78739bcfc3a5b5b21, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/1645bd9057cf4a9c922573079eca9086, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/3c8e928d2d8e424489927535f482d620, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/ecc3d3d5fe32409489544f737312592e, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/f5f76c86ce5f49d98e21b7df754e2b9c, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/c6973cb572fa4b519ca3aaad2a78c689, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/7f01d1fed2d54215b165088dde7c3a49, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/940492fd0c174e0286672200d5b9b2cd, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/221e1ab33201446c9815b31c933a0f71, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/fe6024bfe55a48a29d3eb5e32c7899d8, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/ae71033970e340c9a65b59dc211107e9, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/d16eb543a1154647986050637dac1131, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/8275bd19017d49fc90234317853bf5a1, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/78b890b2565c46888f2b5832bf7cd266, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/cf90fdeba90b46b6862e8535e4ad999a, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/8e110553cd8040b392e868d1ba25330d, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/12c3b3a9efa144de90b54055674e68f5, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/6623073eee49471ab9afce01e7b976ff, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/3a4950b35cde4b9f997883da1dd839cb, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/5f59c5cfe19b4b8bae02f8dee8732980, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/97588e78462f4c89bd775e03002aabe9, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/2a5bdc8ee19a4f81980960dcc74363d6, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/9974ca78841945c79636d441f17a71ac, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/13a724cb76c349a49dc8c74c8dc059f1] to archive 2023-03-31 16:54:09,980 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-31 16:54:09,986 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/010cb846e12645c2ab85d0cc9628bbd3 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/010cb846e12645c2ab85d0cc9628bbd3 2023-03-31 16:54:09,987 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/a19e77acb6394df3a197abed58be738e to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/a19e77acb6394df3a197abed58be738e 2023-03-31 16:54:09,988 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/46b76279f65a48fc99c1179eab1a7971 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/46b76279f65a48fc99c1179eab1a7971 2023-03-31 16:54:09,989 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/ec500428d0c4460dab6c901550e705e4 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/ec500428d0c4460dab6c901550e705e4 2023-03-31 16:54:09,990 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/fa6db4443ac7493a98563caf2a8e7b40 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/fa6db4443ac7493a98563caf2a8e7b40 2023-03-31 16:54:09,992 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/15cfa5e5c30442e2a7602aa3f38a1295 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/15cfa5e5c30442e2a7602aa3f38a1295 2023-03-31 16:54:09,993 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/c447ed0d8629424e93db831f3698ec0e to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/c447ed0d8629424e93db831f3698ec0e 2023-03-31 16:54:09,994 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/07320ee76d544ccea3a912d574b89806 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/07320ee76d544ccea3a912d574b89806 2023-03-31 16:54:09,996 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/68827ed2391e4ca29a272cb933a5cedd to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/68827ed2391e4ca29a272cb933a5cedd 2023-03-31 16:54:09,997 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/ca36732fc7d04a5caee63a13f295f814 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/ca36732fc7d04a5caee63a13f295f814 2023-03-31 16:54:09,998 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/d61f5d3191e243a481f2b953ec45220a to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/d61f5d3191e243a481f2b953ec45220a 2023-03-31 16:54:10,000 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/81256392dfb144279664831b3777c42e to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/81256392dfb144279664831b3777c42e 2023-03-31 16:54:10,001 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/81fd752e0ce5425ba2e00f25e99baeb0 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/81fd752e0ce5425ba2e00f25e99baeb0 2023-03-31 16:54:10,003 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/dba0bdcef83a46b1abacb7a889a8f992 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/dba0bdcef83a46b1abacb7a889a8f992 2023-03-31 16:54:10,004 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/f802b521ea6349c78739bcfc3a5b5b21 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/f802b521ea6349c78739bcfc3a5b5b21 2023-03-31 16:54:10,006 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/1645bd9057cf4a9c922573079eca9086 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/1645bd9057cf4a9c922573079eca9086 2023-03-31 16:54:10,008 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/3c8e928d2d8e424489927535f482d620 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/3c8e928d2d8e424489927535f482d620 2023-03-31 16:54:10,009 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/ecc3d3d5fe32409489544f737312592e to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/ecc3d3d5fe32409489544f737312592e 2023-03-31 16:54:10,010 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/f5f76c86ce5f49d98e21b7df754e2b9c to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/f5f76c86ce5f49d98e21b7df754e2b9c 2023-03-31 16:54:10,012 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/c6973cb572fa4b519ca3aaad2a78c689 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/c6973cb572fa4b519ca3aaad2a78c689 2023-03-31 16:54:10,013 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/7f01d1fed2d54215b165088dde7c3a49 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/7f01d1fed2d54215b165088dde7c3a49 2023-03-31 16:54:10,015 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/940492fd0c174e0286672200d5b9b2cd to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/940492fd0c174e0286672200d5b9b2cd 2023-03-31 16:54:10,024 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/221e1ab33201446c9815b31c933a0f71 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/221e1ab33201446c9815b31c933a0f71 2023-03-31 16:54:10,026 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/fe6024bfe55a48a29d3eb5e32c7899d8 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/fe6024bfe55a48a29d3eb5e32c7899d8 2023-03-31 16:54:10,027 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/ae71033970e340c9a65b59dc211107e9 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/ae71033970e340c9a65b59dc211107e9 2023-03-31 16:54:10,029 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/d16eb543a1154647986050637dac1131 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/d16eb543a1154647986050637dac1131 2023-03-31 16:54:10,030 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/8275bd19017d49fc90234317853bf5a1 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/8275bd19017d49fc90234317853bf5a1 2023-03-31 16:54:10,031 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/78b890b2565c46888f2b5832bf7cd266 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/78b890b2565c46888f2b5832bf7cd266 2023-03-31 16:54:10,032 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/cf90fdeba90b46b6862e8535e4ad999a to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/cf90fdeba90b46b6862e8535e4ad999a 2023-03-31 16:54:10,033 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/8e110553cd8040b392e868d1ba25330d to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/8e110553cd8040b392e868d1ba25330d 2023-03-31 16:54:10,034 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/12c3b3a9efa144de90b54055674e68f5 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/12c3b3a9efa144de90b54055674e68f5 2023-03-31 16:54:10,035 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/6623073eee49471ab9afce01e7b976ff to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/6623073eee49471ab9afce01e7b976ff 2023-03-31 16:54:10,037 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/3a4950b35cde4b9f997883da1dd839cb to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/3a4950b35cde4b9f997883da1dd839cb 2023-03-31 16:54:10,038 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/5f59c5cfe19b4b8bae02f8dee8732980 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/5f59c5cfe19b4b8bae02f8dee8732980 2023-03-31 16:54:10,039 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/97588e78462f4c89bd775e03002aabe9 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/97588e78462f4c89bd775e03002aabe9 2023-03-31 16:54:10,040 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/2a5bdc8ee19a4f81980960dcc74363d6 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/2a5bdc8ee19a4f81980960dcc74363d6 2023-03-31 16:54:10,041 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/9974ca78841945c79636d441f17a71ac to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/9974ca78841945c79636d441f17a71ac 2023-03-31 16:54:10,042 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/13a724cb76c349a49dc8c74c8dc059f1 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/13a724cb76c349a49dc8c74c8dc059f1 2023-03-31 16:54:10,066 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/88d4d41c3eff4a828ff723e2c76501b5, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/ab68f0e33a4f4cd1b8eaccb5ec481708, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/cb55249a2b5148cfb6c0f4445c9c09cb, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/7d5121685cce4e979af87c23f0f6475d, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/1f1c599c73c840b584187ea68eb50fc4, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/e3bd8fd57c824772af50067e773882c9, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/db2a8599b6cb4821bce4873cfd83b9f5, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/921c9b07d4ac47cea568c5511a5fdcc7, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/3fc5cb40778f42ceab1413af62d9ea65, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/4243b53d2d404612b39aa896086ca145, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/ee77cc8f635944a3a75c3485003050e5, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/5176ef7353b34fdd92495e699da94346, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/ad0cf48c84594732a4e59d5d88c010f4, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/d6a8d304ad1f4643918d10cc12956c6a, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/2c04ffc461254f4298677026794e0271, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/1864df0ce2894eb288cbfff2374a9af3, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/6212feb723dc4363a8b87771dd3f4107, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/3eb92bf6351448318f33cec6a39a01bb, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/b11fba94fc2248498af9922a921794cf, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/ed0f0ed9facd4225b2365b8cb81f88b7, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/22eeaae46aaa4aedac30fc05e2a1fcdd, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/3f8bb70b34ba4abbb3f7421ba3682087, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/d6a82a8c0157451eb9050e6a57525e2b, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/0451d5b5df784582b26a1fdf5a5e50c0, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/2d3c5617a87941a6a8b17087da7dcd5d, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/890edee61afc4cc181ba124e76a6f665, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/fd2db7d75aca4434ae6b574dbbb80a07, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/a55c2c3809b14425929ae2608a09ea40, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/ac5fd7c80e094d228f191380eb478244, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/b2cde8036b11485d86cfbdf0994d8025, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/7c4045ff4cb14706a0f47e99b8dba7b7, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/6d9ed6869dc143dfb8fba14ec5a254e5, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/f736406e8e4249369153964ff57a6a2c, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/750df53f91394e0f86227e491812442f, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/893c33bb28f7464289213f4b6ca1bbb7, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/132aca2b84f94772b78de924229fa2c9, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/8afb7f28e8c2411bb5a998d593ae557a, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/a0db6a3e2a554e16aedeedc0d35b2283] to archive 2023-03-31 16:54:10,067 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-31 16:54:10,068 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/88d4d41c3eff4a828ff723e2c76501b5 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/88d4d41c3eff4a828ff723e2c76501b5 2023-03-31 16:54:10,070 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/ab68f0e33a4f4cd1b8eaccb5ec481708 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/ab68f0e33a4f4cd1b8eaccb5ec481708 2023-03-31 16:54:10,071 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/cb55249a2b5148cfb6c0f4445c9c09cb to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/cb55249a2b5148cfb6c0f4445c9c09cb 2023-03-31 16:54:10,072 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/7d5121685cce4e979af87c23f0f6475d to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/7d5121685cce4e979af87c23f0f6475d 2023-03-31 16:54:10,073 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/1f1c599c73c840b584187ea68eb50fc4 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/1f1c599c73c840b584187ea68eb50fc4 2023-03-31 16:54:10,074 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/e3bd8fd57c824772af50067e773882c9 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/e3bd8fd57c824772af50067e773882c9 2023-03-31 16:54:10,076 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/db2a8599b6cb4821bce4873cfd83b9f5 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/db2a8599b6cb4821bce4873cfd83b9f5 2023-03-31 16:54:10,077 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/921c9b07d4ac47cea568c5511a5fdcc7 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/921c9b07d4ac47cea568c5511a5fdcc7 2023-03-31 16:54:10,079 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/3fc5cb40778f42ceab1413af62d9ea65 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/3fc5cb40778f42ceab1413af62d9ea65 2023-03-31 16:54:10,080 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/4243b53d2d404612b39aa896086ca145 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/4243b53d2d404612b39aa896086ca145 2023-03-31 16:54:10,081 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/ee77cc8f635944a3a75c3485003050e5 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/ee77cc8f635944a3a75c3485003050e5 2023-03-31 16:54:10,083 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/5176ef7353b34fdd92495e699da94346 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/5176ef7353b34fdd92495e699da94346 2023-03-31 16:54:10,084 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/ad0cf48c84594732a4e59d5d88c010f4 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/ad0cf48c84594732a4e59d5d88c010f4 2023-03-31 16:54:10,085 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/d6a8d304ad1f4643918d10cc12956c6a to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/d6a8d304ad1f4643918d10cc12956c6a 2023-03-31 16:54:10,086 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/2c04ffc461254f4298677026794e0271 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/2c04ffc461254f4298677026794e0271 2023-03-31 16:54:10,087 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/1864df0ce2894eb288cbfff2374a9af3 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/1864df0ce2894eb288cbfff2374a9af3 2023-03-31 16:54:10,088 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/6212feb723dc4363a8b87771dd3f4107 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/6212feb723dc4363a8b87771dd3f4107 2023-03-31 16:54:10,089 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/3eb92bf6351448318f33cec6a39a01bb to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/3eb92bf6351448318f33cec6a39a01bb 2023-03-31 16:54:10,090 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/b11fba94fc2248498af9922a921794cf to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/b11fba94fc2248498af9922a921794cf 2023-03-31 16:54:10,091 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/ed0f0ed9facd4225b2365b8cb81f88b7 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/ed0f0ed9facd4225b2365b8cb81f88b7 2023-03-31 16:54:10,092 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/22eeaae46aaa4aedac30fc05e2a1fcdd to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/22eeaae46aaa4aedac30fc05e2a1fcdd 2023-03-31 16:54:10,094 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/3f8bb70b34ba4abbb3f7421ba3682087 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/3f8bb70b34ba4abbb3f7421ba3682087 2023-03-31 16:54:10,095 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/d6a82a8c0157451eb9050e6a57525e2b to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/d6a82a8c0157451eb9050e6a57525e2b 2023-03-31 16:54:10,096 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/0451d5b5df784582b26a1fdf5a5e50c0 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/0451d5b5df784582b26a1fdf5a5e50c0 2023-03-31 16:54:10,098 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/2d3c5617a87941a6a8b17087da7dcd5d to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/2d3c5617a87941a6a8b17087da7dcd5d 2023-03-31 16:54:10,099 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/890edee61afc4cc181ba124e76a6f665 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/890edee61afc4cc181ba124e76a6f665 2023-03-31 16:54:10,100 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/fd2db7d75aca4434ae6b574dbbb80a07 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/fd2db7d75aca4434ae6b574dbbb80a07 2023-03-31 16:54:10,101 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/a55c2c3809b14425929ae2608a09ea40 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/a55c2c3809b14425929ae2608a09ea40 2023-03-31 16:54:10,103 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/ac5fd7c80e094d228f191380eb478244 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/ac5fd7c80e094d228f191380eb478244 2023-03-31 16:54:10,104 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/b2cde8036b11485d86cfbdf0994d8025 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/b2cde8036b11485d86cfbdf0994d8025 2023-03-31 16:54:10,105 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/7c4045ff4cb14706a0f47e99b8dba7b7 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/7c4045ff4cb14706a0f47e99b8dba7b7 2023-03-31 16:54:10,106 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/6d9ed6869dc143dfb8fba14ec5a254e5 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/6d9ed6869dc143dfb8fba14ec5a254e5 2023-03-31 16:54:10,108 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/f736406e8e4249369153964ff57a6a2c to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/f736406e8e4249369153964ff57a6a2c 2023-03-31 16:54:10,109 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/750df53f91394e0f86227e491812442f to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/750df53f91394e0f86227e491812442f 2023-03-31 16:54:10,110 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/893c33bb28f7464289213f4b6ca1bbb7 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/893c33bb28f7464289213f4b6ca1bbb7 2023-03-31 16:54:10,111 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/132aca2b84f94772b78de924229fa2c9 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/132aca2b84f94772b78de924229fa2c9 2023-03-31 16:54:10,112 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/8afb7f28e8c2411bb5a998d593ae557a to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/8afb7f28e8c2411bb5a998d593ae557a 2023-03-31 16:54:10,114 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/a0db6a3e2a554e16aedeedc0d35b2283 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/a0db6a3e2a554e16aedeedc0d35b2283 2023-03-31 16:54:10,118 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/8a749e81471b4d64b888c5dc21568044, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/ecd77e38378244408ead7960caf26773, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/8cd7d8ff21044e8e9a36ed332196f6b2, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/adfdf62a70dc4f1b81c8933dc02f819e, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/1ce9b06377fe4597b18ab6b3bcf0ee7c, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/fa8e33de7b824c9a8966ff6a0151e6e4, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/8da43654ffe04f1f9168f9c43e5e3263, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/cccccad0b7854e07bf1f553e86f4675e, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/7474ae0d11f3466786c35e1681db04fb, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/4ca81fd821514dd0833a312accbe5210, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/14ecae34094a42c29e8592284aeeb44f, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/bd6ffc8b9dfc45728b59def07fe6e912, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/c87ddfb34b7f4b49a8a29c3211d6972a, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/22a9c62af3ae44cfb233da38fb3f0b44, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/84f9af705f0043659bad1f6f685eca61, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/52acedccffda4f928340331b4c057a30, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/62110af81f184712ad28ed979415eb2b, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/920249a17bfd4d89bfe479e022083798, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/1011fd0251c1431d92200dd2da9e5622, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/d2d0a538cfd74c61bdd9d06edf1d1fc9, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/1c85f91970d54826961dc4d064feb5a4, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/c9665615bee945e8b1fe6e6e2564c3c1, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/01bb121ed8c048b99bb20915f4faf16d, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/59c1ec4d4c5e4d0aa38c098f66178686, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/13fdc9711b9f4c28bba6cd4307431710, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/55d8160c4f974b8cb369c3aa22d21c9b, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/62602353da6e4ada80784ed765fbd94a, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/bea34f5bbb6a44d28063f8259b6a826d, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/d1eaaaf307824034a64d551410fcc92f, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/2dd6e6fe111a43bab26152d52701c804, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/e16c95940b8049adb4d6fc6634a7b840, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/5c2dbac42bcf4fae9fcead4456abe051, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/5a42f3222f8d460687b1f6da4283918f, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/e3776ff91f63473dab8744134928d105, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/66ba1c9fbf04488eabf00d167833ae72, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/82a8878f3ac943c3806450ed60be9e5c, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/a789411c10b5444ea2ee03e40e33e17a] to archive 2023-03-31 16:54:10,119 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-31 16:54:10,120 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/8a749e81471b4d64b888c5dc21568044 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/8a749e81471b4d64b888c5dc21568044 2023-03-31 16:54:10,121 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/ecd77e38378244408ead7960caf26773 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/ecd77e38378244408ead7960caf26773 2023-03-31 16:54:10,122 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/8cd7d8ff21044e8e9a36ed332196f6b2 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/8cd7d8ff21044e8e9a36ed332196f6b2 2023-03-31 16:54:10,123 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/adfdf62a70dc4f1b81c8933dc02f819e to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/adfdf62a70dc4f1b81c8933dc02f819e 2023-03-31 16:54:10,124 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/1ce9b06377fe4597b18ab6b3bcf0ee7c to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/1ce9b06377fe4597b18ab6b3bcf0ee7c 2023-03-31 16:54:10,125 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/fa8e33de7b824c9a8966ff6a0151e6e4 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/fa8e33de7b824c9a8966ff6a0151e6e4 2023-03-31 16:54:10,126 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/8da43654ffe04f1f9168f9c43e5e3263 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/8da43654ffe04f1f9168f9c43e5e3263 2023-03-31 16:54:10,127 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/cccccad0b7854e07bf1f553e86f4675e to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/cccccad0b7854e07bf1f553e86f4675e 2023-03-31 16:54:10,128 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/7474ae0d11f3466786c35e1681db04fb to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/7474ae0d11f3466786c35e1681db04fb 2023-03-31 16:54:10,129 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/4ca81fd821514dd0833a312accbe5210 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/4ca81fd821514dd0833a312accbe5210 2023-03-31 16:54:10,130 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/14ecae34094a42c29e8592284aeeb44f to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/14ecae34094a42c29e8592284aeeb44f 2023-03-31 16:54:10,132 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/bd6ffc8b9dfc45728b59def07fe6e912 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/bd6ffc8b9dfc45728b59def07fe6e912 2023-03-31 16:54:10,133 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/c87ddfb34b7f4b49a8a29c3211d6972a to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/c87ddfb34b7f4b49a8a29c3211d6972a 2023-03-31 16:54:10,135 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/22a9c62af3ae44cfb233da38fb3f0b44 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/22a9c62af3ae44cfb233da38fb3f0b44 2023-03-31 16:54:10,137 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/84f9af705f0043659bad1f6f685eca61 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/84f9af705f0043659bad1f6f685eca61 2023-03-31 16:54:10,138 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/52acedccffda4f928340331b4c057a30 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/52acedccffda4f928340331b4c057a30 2023-03-31 16:54:10,140 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/62110af81f184712ad28ed979415eb2b to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/62110af81f184712ad28ed979415eb2b 2023-03-31 16:54:10,141 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/920249a17bfd4d89bfe479e022083798 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/920249a17bfd4d89bfe479e022083798 2023-03-31 16:54:10,143 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/1011fd0251c1431d92200dd2da9e5622 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/1011fd0251c1431d92200dd2da9e5622 2023-03-31 16:54:10,144 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/d2d0a538cfd74c61bdd9d06edf1d1fc9 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/d2d0a538cfd74c61bdd9d06edf1d1fc9 2023-03-31 16:54:10,146 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/1c85f91970d54826961dc4d064feb5a4 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/1c85f91970d54826961dc4d064feb5a4 2023-03-31 16:54:10,148 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/c9665615bee945e8b1fe6e6e2564c3c1 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/c9665615bee945e8b1fe6e6e2564c3c1 2023-03-31 16:54:10,149 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/01bb121ed8c048b99bb20915f4faf16d to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/01bb121ed8c048b99bb20915f4faf16d 2023-03-31 16:54:10,150 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/59c1ec4d4c5e4d0aa38c098f66178686 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/59c1ec4d4c5e4d0aa38c098f66178686 2023-03-31 16:54:10,152 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/13fdc9711b9f4c28bba6cd4307431710 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/13fdc9711b9f4c28bba6cd4307431710 2023-03-31 16:54:10,153 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/55d8160c4f974b8cb369c3aa22d21c9b to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/55d8160c4f974b8cb369c3aa22d21c9b 2023-03-31 16:54:10,154 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/62602353da6e4ada80784ed765fbd94a to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/62602353da6e4ada80784ed765fbd94a 2023-03-31 16:54:10,156 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/bea34f5bbb6a44d28063f8259b6a826d to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/bea34f5bbb6a44d28063f8259b6a826d 2023-03-31 16:54:10,157 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/d1eaaaf307824034a64d551410fcc92f to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/d1eaaaf307824034a64d551410fcc92f 2023-03-31 16:54:10,158 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/2dd6e6fe111a43bab26152d52701c804 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/2dd6e6fe111a43bab26152d52701c804 2023-03-31 16:54:10,159 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/e16c95940b8049adb4d6fc6634a7b840 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/e16c95940b8049adb4d6fc6634a7b840 2023-03-31 16:54:10,161 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/5c2dbac42bcf4fae9fcead4456abe051 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/5c2dbac42bcf4fae9fcead4456abe051 2023-03-31 16:54:10,162 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/5a42f3222f8d460687b1f6da4283918f to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/5a42f3222f8d460687b1f6da4283918f 2023-03-31 16:54:10,164 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/e3776ff91f63473dab8744134928d105 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/e3776ff91f63473dab8744134928d105 2023-03-31 16:54:10,165 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/66ba1c9fbf04488eabf00d167833ae72 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/66ba1c9fbf04488eabf00d167833ae72 2023-03-31 16:54:10,167 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/82a8878f3ac943c3806450ed60be9e5c to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/82a8878f3ac943c3806450ed60be9e5c 2023-03-31 16:54:10,168 DEBUG [StoreCloser-TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/a789411c10b5444ea2ee03e40e33e17a to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/a789411c10b5444ea2ee03e40e33e17a 2023-03-31 16:54:10,176 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/recovered.edits/969.seqid, newMaxSeqId=969, maxSeqId=1 2023-03-31 16:54:10,177 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4. 2023-03-31 16:54:10,177 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for e94df50bb46b68756c6a704332e1c5f4: 2023-03-31 16:54:10,179 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed e94df50bb46b68756c6a704332e1c5f4 2023-03-31 16:54:10,179 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=e94df50bb46b68756c6a704332e1c5f4, regionState=CLOSED 2023-03-31 16:54:10,180 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680281650179"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680281650179"}]},"ts":"1680281650179"} 2023-03-31 16:54:10,182 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-03-31 16:54:10,182 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; CloseRegionProcedure e94df50bb46b68756c6a704332e1c5f4, server=jenkins-hbase4.apache.org,33441,1680281619375 in 1.6280 sec 2023-03-31 16:54:10,183 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-03-31 16:54:10,183 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=e94df50bb46b68756c6a704332e1c5f4, UNASSIGN in 1.6320 sec 2023-03-31 16:54:10,184 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680281650184"}]},"ts":"1680281650184"} 2023-03-31 16:54:10,185 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-31 16:54:10,186 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-31 16:54:10,188 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.6490 sec 2023-03-31 16:54:10,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-31 16:54:10,651 INFO [Listener at localhost/44135] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 12 completed 2023-03-31 16:54:10,655 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-03-31 16:54:10,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure2.ProcedureExecutor(1029): Stored pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-31 16:54:10,663 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-31 16:54:10,665 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-31 16:54:10,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-03-31 16:54:10,669 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4 2023-03-31 16:54:10,671 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A, FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B, FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C, FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/recovered.edits] 2023-03-31 16:54:10,674 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/c61f28c2e95c4dde9a0cb46609ef5208 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/c61f28c2e95c4dde9a0cb46609ef5208 2023-03-31 16:54:10,675 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/d881ce8497e044ed882a99d2520ca34b to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/A/d881ce8497e044ed882a99d2520ca34b 2023-03-31 16:54:10,678 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/59b0f8c6e2d94198a9be44d67304f04c to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/59b0f8c6e2d94198a9be44d67304f04c 2023-03-31 16:54:10,679 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/b46068808d414eeb9f901cb44cf490b0 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/B/b46068808d414eeb9f901cb44cf490b0 2023-03-31 16:54:10,681 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/5a96250256424f1bac22e9e1925e8d98 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/5a96250256424f1bac22e9e1925e8d98 2023-03-31 16:54:10,682 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/e3d1fcf2e082486988df6496a1bd0813 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/C/e3d1fcf2e082486988df6496a1bd0813 2023-03-31 16:54:10,685 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/recovered.edits/969.seqid to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4/recovered.edits/969.seqid 2023-03-31 16:54:10,685 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/e94df50bb46b68756c6a704332e1c5f4 2023-03-31 16:54:10,685 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-31 16:54:10,688 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-31 16:54:10,708 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-31 16:54:10,749 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-31 16:54:10,751 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-31 16:54:10,751 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-31 16:54:10,751 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680281650751"}]},"ts":"9223372036854775807"} 2023-03-31 16:54:10,753 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-31 16:54:10,753 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => e94df50bb46b68756c6a704332e1c5f4, NAME => 'TestAcidGuarantees,,1680281623312.e94df50bb46b68756c6a704332e1c5f4.', STARTKEY => '', ENDKEY => ''}] 2023-03-31 16:54:10,753 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-31 16:54:10,753 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680281650753"}]},"ts":"9223372036854775807"} 2023-03-31 16:54:10,755 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-31 16:54:10,757 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-31 16:54:10,758 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=15, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 101 msec 2023-03-31 16:54:10,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-03-31 16:54:10,768 INFO [Listener at localhost/44135] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 15 completed 2023-03-31 16:54:10,786 INFO [Listener at localhost/44135] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=232 (was 206) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x56ede43c-shared-pool-18 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: (jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:0;jenkins-hbase4:33441-shortCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-12 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-13 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x56ede43c-shared-pool-15 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x56ede43c-shared-pool-16 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x56ede43c-shared-pool-17 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-11 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_956352929_15 at /127.0.0.1:42060 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=581 (was 470) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=371 (was 333) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=3826 (was 4444) 2023-03-31 16:54:10,798 INFO [Listener at localhost/44135] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=232, OpenFileDescriptor=581, MaxFileDescriptor=60000, SystemLoadAverage=371, ProcessCount=169, AvailableMemoryMB=3825 2023-03-31 16:54:10,799 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-31 16:54:10,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-31 16:54:10,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure2.ProcedureExecutor(1029): Stored pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-31 16:54:10,802 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-31 16:54:10,802 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 16 2023-03-31 16:54:10,803 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-31 16:54:10,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-31 16:54:10,804 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b 2023-03-31 16:54:10,805 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b empty. 2023-03-31 16:54:10,805 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b 2023-03-31 16:54:10,805 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-31 16:54:10,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-31 16:54:11,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-31 16:54:11,214 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-31 16:54:11,217 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => e76385ddbd37af190ba2df92b5602e3b, NAME => 'TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp 2023-03-31 16:54:11,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-31 16:54:11,624 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-31 16:54:11,624 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing e76385ddbd37af190ba2df92b5602e3b, disabling compactions & flushes 2023-03-31 16:54:11,625 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:11,625 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:11,625 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. after waiting 0 ms 2023-03-31 16:54:11,625 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:11,625 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:11,625 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for e76385ddbd37af190ba2df92b5602e3b: 2023-03-31 16:54:11,628 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-31 16:54:11,629 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680281651629"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680281651629"}]},"ts":"1680281651629"} 2023-03-31 16:54:11,631 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-31 16:54:11,632 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-31 16:54:11,632 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680281651632"}]},"ts":"1680281651632"} 2023-03-31 16:54:11,633 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-31 16:54:11,638 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e76385ddbd37af190ba2df92b5602e3b, ASSIGN}] 2023-03-31 16:54:11,642 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e76385ddbd37af190ba2df92b5602e3b, ASSIGN 2023-03-31 16:54:11,642 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=e76385ddbd37af190ba2df92b5602e3b, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,33441,1680281619375; forceNewPlan=false, retain=false 2023-03-31 16:54:11,794 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=e76385ddbd37af190ba2df92b5602e3b, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:11,794 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680281651793"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680281651793"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680281651793"}]},"ts":"1680281651793"} 2023-03-31 16:54:11,796 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE; OpenRegionProcedure e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375}] 2023-03-31 16:54:11,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-31 16:54:11,951 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:11,951 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => e76385ddbd37af190ba2df92b5602e3b, NAME => 'TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.', STARTKEY => '', ENDKEY => ''} 2023-03-31 16:54:11,951 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees e76385ddbd37af190ba2df92b5602e3b 2023-03-31 16:54:11,952 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-31 16:54:11,952 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for e76385ddbd37af190ba2df92b5602e3b 2023-03-31 16:54:11,952 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for e76385ddbd37af190ba2df92b5602e3b 2023-03-31 16:54:11,953 INFO [StoreOpener-e76385ddbd37af190ba2df92b5602e3b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region e76385ddbd37af190ba2df92b5602e3b 2023-03-31 16:54:11,954 DEBUG [StoreOpener-e76385ddbd37af190ba2df92b5602e3b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A 2023-03-31 16:54:11,954 DEBUG [StoreOpener-e76385ddbd37af190ba2df92b5602e3b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A 2023-03-31 16:54:11,955 INFO [StoreOpener-e76385ddbd37af190ba2df92b5602e3b-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-31 16:54:11,955 INFO [StoreOpener-e76385ddbd37af190ba2df92b5602e3b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e76385ddbd37af190ba2df92b5602e3b columnFamilyName A 2023-03-31 16:54:11,955 INFO [StoreOpener-e76385ddbd37af190ba2df92b5602e3b-1] regionserver.HStore(310): Store=e76385ddbd37af190ba2df92b5602e3b/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-31 16:54:11,956 INFO [StoreOpener-e76385ddbd37af190ba2df92b5602e3b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region e76385ddbd37af190ba2df92b5602e3b 2023-03-31 16:54:11,957 DEBUG [StoreOpener-e76385ddbd37af190ba2df92b5602e3b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B 2023-03-31 16:54:11,957 DEBUG [StoreOpener-e76385ddbd37af190ba2df92b5602e3b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B 2023-03-31 16:54:11,957 INFO [StoreOpener-e76385ddbd37af190ba2df92b5602e3b-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-31 16:54:11,957 INFO [StoreOpener-e76385ddbd37af190ba2df92b5602e3b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e76385ddbd37af190ba2df92b5602e3b columnFamilyName B 2023-03-31 16:54:11,957 INFO [StoreOpener-e76385ddbd37af190ba2df92b5602e3b-1] regionserver.HStore(310): Store=e76385ddbd37af190ba2df92b5602e3b/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-31 16:54:11,958 INFO [StoreOpener-e76385ddbd37af190ba2df92b5602e3b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region e76385ddbd37af190ba2df92b5602e3b 2023-03-31 16:54:11,959 DEBUG [StoreOpener-e76385ddbd37af190ba2df92b5602e3b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C 2023-03-31 16:54:11,959 DEBUG [StoreOpener-e76385ddbd37af190ba2df92b5602e3b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C 2023-03-31 16:54:11,959 INFO [StoreOpener-e76385ddbd37af190ba2df92b5602e3b-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-31 16:54:11,959 INFO [StoreOpener-e76385ddbd37af190ba2df92b5602e3b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e76385ddbd37af190ba2df92b5602e3b columnFamilyName C 2023-03-31 16:54:11,959 INFO [StoreOpener-e76385ddbd37af190ba2df92b5602e3b-1] regionserver.HStore(310): Store=e76385ddbd37af190ba2df92b5602e3b/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-31 16:54:11,959 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:11,960 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b 2023-03-31 16:54:11,961 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b 2023-03-31 16:54:11,962 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-31 16:54:11,963 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for e76385ddbd37af190ba2df92b5602e3b 2023-03-31 16:54:11,965 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-31 16:54:11,966 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened e76385ddbd37af190ba2df92b5602e3b; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59627455, jitterRate=-0.1114816814661026}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-31 16:54:11,966 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for e76385ddbd37af190ba2df92b5602e3b: 2023-03-31 16:54:11,966 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b., pid=18, masterSystemTime=1680281651947 2023-03-31 16:54:11,968 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:11,968 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:11,968 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=e76385ddbd37af190ba2df92b5602e3b, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:11,968 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680281651968"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680281651968"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680281651968"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680281651968"}]},"ts":"1680281651968"} 2023-03-31 16:54:11,972 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-03-31 16:54:11,972 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; OpenRegionProcedure e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 in 175 msec 2023-03-31 16:54:11,974 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=16 2023-03-31 16:54:11,974 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=16, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=e76385ddbd37af190ba2df92b5602e3b, ASSIGN in 334 msec 2023-03-31 16:54:11,974 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-31 16:54:11,975 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680281651974"}]},"ts":"1680281651974"} 2023-03-31 16:54:11,975 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-31 16:54:11,977 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-31 16:54:11,979 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=16, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1770 sec 2023-03-31 16:54:12,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-31 16:54:12,909 INFO [Listener at localhost/44135] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 16 completed 2023-03-31 16:54:12,910 DEBUG [Listener at localhost/44135] zookeeper.ReadOnlyZKClient(139): Connect 0x5b8d0eb2 to 127.0.0.1:51916 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 16:54:12,914 DEBUG [Listener at localhost/44135] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@52274650, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 16:54:12,916 DEBUG [Listener at localhost/44135] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 16:54:12,920 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33392, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 16:54:12,921 DEBUG [Listener at localhost/44135] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-31 16:54:12,924 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51268, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-31 16:54:12,932 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-31 16:54:12,933 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-31 16:54:12,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure2.ProcedureExecutor(1029): Stored pid=19, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-03-31 16:54:13,359 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-31 16:54:13,361 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-31 16:54:13,361 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-31 16:54:13,366 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=19, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-03-31 16:54:13,378 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e76385ddbd37af190ba2df92b5602e3b, REOPEN/MOVE}] 2023-03-31 16:54:13,379 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e76385ddbd37af190ba2df92b5602e3b, REOPEN/MOVE 2023-03-31 16:54:13,380 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=e76385ddbd37af190ba2df92b5602e3b, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:13,380 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680281653380"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680281653380"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680281653380"}]},"ts":"1680281653380"} 2023-03-31 16:54:13,381 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=21, state=RUNNABLE; CloseRegionProcedure e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375}] 2023-03-31 16:54:13,533 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close e76385ddbd37af190ba2df92b5602e3b 2023-03-31 16:54:13,534 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing e76385ddbd37af190ba2df92b5602e3b, disabling compactions & flushes 2023-03-31 16:54:13,534 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:13,534 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:13,534 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. after waiting 0 ms 2023-03-31 16:54:13,535 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:13,538 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-03-31 16:54:13,539 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:13,539 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for e76385ddbd37af190ba2df92b5602e3b: 2023-03-31 16:54:13,539 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: e76385ddbd37af190ba2df92b5602e3b to self. 2023-03-31 16:54:13,541 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed e76385ddbd37af190ba2df92b5602e3b 2023-03-31 16:54:13,541 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=e76385ddbd37af190ba2df92b5602e3b, regionState=CLOSED 2023-03-31 16:54:13,541 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680281653541"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680281653541"}]},"ts":"1680281653541"} 2023-03-31 16:54:13,544 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=21 2023-03-31 16:54:13,544 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=21, state=SUCCESS; CloseRegionProcedure e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 in 162 msec 2023-03-31 16:54:13,545 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=e76385ddbd37af190ba2df92b5602e3b, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,33441,1680281619375; forceNewPlan=false, retain=true 2023-03-31 16:54:13,695 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=e76385ddbd37af190ba2df92b5602e3b, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:13,696 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680281653695"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680281653695"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680281653695"}]},"ts":"1680281653695"} 2023-03-31 16:54:13,698 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=21, state=RUNNABLE; OpenRegionProcedure e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375}] 2023-03-31 16:54:13,853 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:13,853 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => e76385ddbd37af190ba2df92b5602e3b, NAME => 'TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.', STARTKEY => '', ENDKEY => ''} 2023-03-31 16:54:13,853 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees e76385ddbd37af190ba2df92b5602e3b 2023-03-31 16:54:13,853 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-31 16:54:13,853 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for e76385ddbd37af190ba2df92b5602e3b 2023-03-31 16:54:13,853 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for e76385ddbd37af190ba2df92b5602e3b 2023-03-31 16:54:13,856 INFO [StoreOpener-e76385ddbd37af190ba2df92b5602e3b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region e76385ddbd37af190ba2df92b5602e3b 2023-03-31 16:54:13,857 DEBUG [StoreOpener-e76385ddbd37af190ba2df92b5602e3b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A 2023-03-31 16:54:13,857 DEBUG [StoreOpener-e76385ddbd37af190ba2df92b5602e3b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A 2023-03-31 16:54:13,857 INFO [StoreOpener-e76385ddbd37af190ba2df92b5602e3b-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-31 16:54:13,859 INFO [StoreOpener-e76385ddbd37af190ba2df92b5602e3b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e76385ddbd37af190ba2df92b5602e3b columnFamilyName A 2023-03-31 16:54:13,861 INFO [StoreOpener-e76385ddbd37af190ba2df92b5602e3b-1] regionserver.HStore(310): Store=e76385ddbd37af190ba2df92b5602e3b/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-31 16:54:13,862 INFO [StoreOpener-e76385ddbd37af190ba2df92b5602e3b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region e76385ddbd37af190ba2df92b5602e3b 2023-03-31 16:54:13,863 DEBUG [StoreOpener-e76385ddbd37af190ba2df92b5602e3b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B 2023-03-31 16:54:13,863 DEBUG [StoreOpener-e76385ddbd37af190ba2df92b5602e3b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B 2023-03-31 16:54:13,863 INFO [StoreOpener-e76385ddbd37af190ba2df92b5602e3b-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-31 16:54:13,863 INFO [StoreOpener-e76385ddbd37af190ba2df92b5602e3b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e76385ddbd37af190ba2df92b5602e3b columnFamilyName B 2023-03-31 16:54:13,864 INFO [StoreOpener-e76385ddbd37af190ba2df92b5602e3b-1] regionserver.HStore(310): Store=e76385ddbd37af190ba2df92b5602e3b/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-31 16:54:13,864 INFO [StoreOpener-e76385ddbd37af190ba2df92b5602e3b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region e76385ddbd37af190ba2df92b5602e3b 2023-03-31 16:54:13,865 DEBUG [StoreOpener-e76385ddbd37af190ba2df92b5602e3b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C 2023-03-31 16:54:13,865 DEBUG [StoreOpener-e76385ddbd37af190ba2df92b5602e3b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C 2023-03-31 16:54:13,865 INFO [StoreOpener-e76385ddbd37af190ba2df92b5602e3b-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-31 16:54:13,866 INFO [StoreOpener-e76385ddbd37af190ba2df92b5602e3b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e76385ddbd37af190ba2df92b5602e3b columnFamilyName C 2023-03-31 16:54:13,866 INFO [StoreOpener-e76385ddbd37af190ba2df92b5602e3b-1] regionserver.HStore(310): Store=e76385ddbd37af190ba2df92b5602e3b/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-31 16:54:13,866 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:13,867 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b 2023-03-31 16:54:13,868 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b 2023-03-31 16:54:13,870 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-31 16:54:13,871 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for e76385ddbd37af190ba2df92b5602e3b 2023-03-31 16:54:13,872 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened e76385ddbd37af190ba2df92b5602e3b; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63574771, jitterRate=-0.052662089467048645}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-31 16:54:13,872 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for e76385ddbd37af190ba2df92b5602e3b: 2023-03-31 16:54:13,872 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b., pid=23, masterSystemTime=1680281653849 2023-03-31 16:54:13,874 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:13,874 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:13,874 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=e76385ddbd37af190ba2df92b5602e3b, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:13,875 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680281653874"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680281653874"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680281653874"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680281653874"}]},"ts":"1680281653874"} 2023-03-31 16:54:13,877 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=21 2023-03-31 16:54:13,877 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=21, state=SUCCESS; OpenRegionProcedure e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 in 178 msec 2023-03-31 16:54:13,879 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-03-31 16:54:13,879 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=e76385ddbd37af190ba2df92b5602e3b, REOPEN/MOVE in 499 msec 2023-03-31 16:54:13,881 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=19 2023-03-31 16:54:13,881 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=19, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 516 msec 2023-03-31 16:54:13,883 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=19, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 945 msec 2023-03-31 16:54:13,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1227): Checking to see if procedure is done pid=19 2023-03-31 16:54:13,891 DEBUG [Listener at localhost/44135] zookeeper.ReadOnlyZKClient(139): Connect 0x051c62b6 to 127.0.0.1:51916 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 16:54:13,897 DEBUG [Listener at localhost/44135] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@21281a37, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 16:54:13,898 DEBUG [Listener at localhost/44135] zookeeper.ReadOnlyZKClient(139): Connect 0x0a71edf0 to 127.0.0.1:51916 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 16:54:13,901 DEBUG [Listener at localhost/44135] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2bab6b2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 16:54:13,901 DEBUG [Listener at localhost/44135] zookeeper.ReadOnlyZKClient(139): Connect 0x21265fcb to 127.0.0.1:51916 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 16:54:13,905 DEBUG [Listener at localhost/44135] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@24bd3872, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 16:54:13,906 DEBUG [Listener at localhost/44135] zookeeper.ReadOnlyZKClient(139): Connect 0x7aba8e46 to 127.0.0.1:51916 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 16:54:13,911 DEBUG [Listener at localhost/44135] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@370947cb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 16:54:13,911 DEBUG [Listener at localhost/44135] zookeeper.ReadOnlyZKClient(139): Connect 0x0feb770d to 127.0.0.1:51916 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 16:54:13,916 DEBUG [Listener at localhost/44135] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@210d1008, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 16:54:13,917 DEBUG [Listener at localhost/44135] zookeeper.ReadOnlyZKClient(139): Connect 0x00c2598b to 127.0.0.1:51916 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 16:54:13,922 DEBUG [Listener at localhost/44135] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@498694cb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 16:54:13,922 DEBUG [Listener at localhost/44135] zookeeper.ReadOnlyZKClient(139): Connect 0x508246cc to 127.0.0.1:51916 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 16:54:13,926 DEBUG [Listener at localhost/44135] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@421e5356, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 16:54:13,926 DEBUG [Listener at localhost/44135] zookeeper.ReadOnlyZKClient(139): Connect 0x08795451 to 127.0.0.1:51916 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 16:54:13,930 DEBUG [Listener at localhost/44135] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4a0fd7b4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 16:54:13,930 DEBUG [Listener at localhost/44135] zookeeper.ReadOnlyZKClient(139): Connect 0x58e27a32 to 127.0.0.1:51916 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 16:54:13,936 DEBUG [Listener at localhost/44135] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1a7b66a3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 16:54:13,948 DEBUG [hconnection-0x6bfe7acc-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 16:54:13,948 DEBUG [hconnection-0x958923d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 16:54:13,950 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52864, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 16:54:13,953 DEBUG [hconnection-0x375264ac-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 16:54:13,953 DEBUG [hconnection-0xad060d3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 16:54:13,954 DEBUG [hconnection-0x7f5a7560-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 16:54:13,956 DEBUG [hconnection-0x25ae36a0-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 16:54:13,962 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52882, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 16:54:13,962 DEBUG [hconnection-0x27d54997-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 16:54:13,963 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52878, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 16:54:13,963 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52872, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 16:54:13,963 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52906, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 16:54:13,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:13,964 DEBUG [hconnection-0x604018af-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 16:54:13,966 DEBUG [hconnection-0x7a52e68c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 16:54:13,972 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52892, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 16:54:13,972 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52888, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 16:54:13,972 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52934, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 16:54:13,972 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52920, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 16:54:13,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:13,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:13,988 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:13,989 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:13,989 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:13,989 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:13,989 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:13,989 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:13,993 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:13,993 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:13,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on e76385ddbd37af190ba2df92b5602e3b 2023-03-31 16:54:13,994 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e76385ddbd37af190ba2df92b5602e3b 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-31 16:54:13,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:13,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:13,995 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:13,995 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:13,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:13,995 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e76385ddbd37af190ba2df92b5602e3b, store=A 2023-03-31 16:54:13,995 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:13,995 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e76385ddbd37af190ba2df92b5602e3b, store=B 2023-03-31 16:54:13,995 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:13,995 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:13,996 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e76385ddbd37af190ba2df92b5602e3b, store=C 2023-03-31 16:54:13,996 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:13,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:13,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:13,996 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:13,997 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:13,997 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:13,997 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:13,997 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:13,997 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:13,997 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:13,997 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:13,997 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:13,998 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. started... 2023-03-31 16:54:13,998 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. as already flushing 2023-03-31 16:54:13,998 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:14,000 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:14,000 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:14,000 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,003 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:14,003 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:14,004 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:14,004 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:14,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:14,005 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:14,005 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:14,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:14,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:14,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:14,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:14,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:14,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@388f2f03[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:14,007 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:14,007 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:14,008 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:14,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:14,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:14,009 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:14,009 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,009 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:14,009 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:14,009 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:14,011 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,011 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:14,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,011 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:14,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:14,011 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:14,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:14,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:14,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:14,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:14,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:14,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:14,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:14,015 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:14,015 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:14,015 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:14,015 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:14,015 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:14,015 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:14,017 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:14,017 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:14,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:14,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:14,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:14,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:14,017 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:14,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:14,017 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:14,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:14,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:14,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:14,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:14,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:14,019 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:14,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:14,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:14,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,022 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:14,022 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:14,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:14,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:14,022 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,022 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:14,023 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:14,023 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,023 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:14,023 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:14,023 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:14,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:14,023 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:14,023 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:14,023 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:14,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:14,023 DEBUG [Thread-907] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:14,023 DEBUG [Thread-907] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:14,023 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:14,034 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:14,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52906 deadline: 1680281714029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,036 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680281654024, max timestamp=1680281654033Num uniques 859; 2023-03-31 16:54:14,036 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680281654024, max timestamp=1680281654033Num uniques 859; 2023-03-31 16:54:14,036 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-31 16:54:14,038 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680281654024, max timestamp=1680281654033Num uniques 859; 2023-03-31 16:54:14,039 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-31 16:54:14,036 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-31 16:54:14,042 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230331ed1519f576a94a9f8b7866a74c5f2475 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331ed1519f576a94a9f8b7866a74c5f2475 2023-03-31 16:54:14,056 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:14,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52920 deadline: 1680281714053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,057 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:14,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52892 deadline: 1680281714054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,057 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:14,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52878 deadline: 1680281714056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,058 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:14,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52872 deadline: 1680281714057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,072 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/A/22abf05162964cb698f718f8d8a09162 2023-03-31 16:54:14,104 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/B/bb3f5d3f50584c91a07b0498a7d638eb 2023-03-31 16:54:14,123 DEBUG [Thread-907] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:14,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:14,127 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:14,140 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:14,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52906 deadline: 1680281714140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:14,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:14,152 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:14,152 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:14,152 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:14,152 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:14,153 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:14,153 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:14,154 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:14,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:14,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:14,154 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:14,155 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,155 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:14,155 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:14,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:14,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:14,156 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:14,156 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:14,156 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:14,156 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:14,156 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:14,156 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:14,161 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:14,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52920 deadline: 1680281714158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,161 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:14,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52878 deadline: 1680281714159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,162 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:14,162 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:14,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52892 deadline: 1680281714159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52872 deadline: 1680281714160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,168 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:14,168 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:14,168 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:14,168 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. started... 2023-03-31 16:54:14,168 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. as already flushing 2023-03-31 16:54:14,168 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:14,168 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:14,168 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:14,168 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,172 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:14,172 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:14,175 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:14,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:14,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:14,176 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:14,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:14,176 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:14,176 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:14,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:14,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:14,178 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:14,179 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2543c86[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:14,179 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:14,179 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:14,180 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:14,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:14,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:14,180 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:14,180 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:14,180 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:14,180 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,180 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:14,181 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:14,182 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:14,182 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:14,182 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:14,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:14,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:14,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:14,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:14,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:14,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:14,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:14,186 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:14,186 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:14,186 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:14,186 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:14,186 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:14,186 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:14,188 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:14,188 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:14,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:14,188 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:14,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:14,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:14,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:14,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:14,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:14,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:14,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:14,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:14,189 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:14,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:14,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:14,191 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:14,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:14,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,194 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,194 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:14,194 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:14,194 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:14,194 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:14,194 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:14,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:14,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:14,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:14,194 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:14,195 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,195 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:14,195 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:14,195 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:14,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:14,195 DEBUG [Thread-907] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:14,195 DEBUG [Thread-907] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:14,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:14,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:14,295 DEBUG [Thread-907] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:14,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:14,299 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:14,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:14,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:14,308 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:14,308 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:14,308 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:14,308 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:14,309 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:14,309 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:14,310 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:14,310 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:14,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:14,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:14,311 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:14,311 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,312 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:14,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:14,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:14,312 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:14,312 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:14,312 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:14,313 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:14,313 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:14,313 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:14,314 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:14,314 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:14,314 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:14,314 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. started... 2023-03-31 16:54:14,314 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. as already flushing 2023-03-31 16:54:14,314 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:14,314 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:14,315 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:14,315 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,316 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:14,316 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:14,318 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:14,318 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:14,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:14,318 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:14,318 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:14,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:14,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:14,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:14,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:14,320 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:14,320 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@98109[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:14,320 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:14,320 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:14,323 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,323 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:14,329 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:14,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:14,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:14,329 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:14,329 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:14,329 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:14,331 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:14,331 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,331 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:14,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:14,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:14,331 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:14,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:14,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:14,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:14,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:14,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:14,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:14,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:14,333 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:14,333 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:14,333 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:14,333 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:14,334 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:14,335 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:14,335 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:14,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:14,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:14,335 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:14,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:14,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:14,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:14,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:14,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:14,336 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:14,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:14,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:14,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:14,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:14,342 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:14,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:14,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,344 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:14,344 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:14,344 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,344 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:14,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:14,344 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:14,344 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:14,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:14,344 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:14,344 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:14,344 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:14,345 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,345 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:14,345 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:14,345 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:14,345 DEBUG [Thread-907] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:14,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:14,345 DEBUG [Thread-907] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:14,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:14,347 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:14,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52906 deadline: 1680281714343, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,365 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:14,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52920 deadline: 1680281714363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,365 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:14,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52892 deadline: 1680281714364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,365 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:14,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52878 deadline: 1680281714364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,366 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:14,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52872 deadline: 1680281714364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,445 DEBUG [Thread-907] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:14,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:14,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:14,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:14,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:14,459 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:14,459 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:14,459 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:14,459 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:14,459 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:14,459 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:14,463 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:14,463 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:14,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:14,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:14,464 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,464 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:14,464 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:14,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:14,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:14,464 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:14,480 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:14,480 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:14,480 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:14,480 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:14,480 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:14,480 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:14,480 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:14,480 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. started... 2023-03-31 16:54:14,480 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:14,481 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. as already flushing 2023-03-31 16:54:14,481 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:14,481 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:14,481 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:14,481 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,481 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:14,481 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:14,485 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,485 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:14,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:14,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:14,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:14,486 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:14,486 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:14,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:14,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:14,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:14,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:14,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d6e6af3[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:14,488 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:14,488 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:14,489 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:14,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:14,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:14,489 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:14,489 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:14,489 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:14,489 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,490 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:14,491 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:14,491 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:14,491 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:14,492 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:14,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:14,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:14,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:14,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:14,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:14,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:14,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:14,495 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:14,495 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:14,496 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:14,496 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:14,496 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:14,496 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:14,499 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:14,499 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:14,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:14,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:14,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:14,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:14,499 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:14,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:14,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:14,499 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:14,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:14,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:14,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:14,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:14,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:14,501 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:14,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:14,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,503 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:14,503 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,503 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:14,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:14,503 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:14,503 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:14,503 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:14,504 DEBUG [Thread-907] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:14,504 DEBUG [Thread-907] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:14,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:14,503 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:14,503 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:14,504 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:14,504 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,504 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:14,504 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:14,504 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:14,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:14,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:14,532 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/C/308dfc7bca3948e690cefe3994e99656 2023-03-31 16:54:14,541 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/A/22abf05162964cb698f718f8d8a09162 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/22abf05162964cb698f718f8d8a09162 2023-03-31 16:54:14,548 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/22abf05162964cb698f718f8d8a09162, entries=200, sequenceid=15, filesize=31.9 K 2023-03-31 16:54:14,549 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/B/bb3f5d3f50584c91a07b0498a7d638eb as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/bb3f5d3f50584c91a07b0498a7d638eb 2023-03-31 16:54:14,558 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/bb3f5d3f50584c91a07b0498a7d638eb, entries=150, sequenceid=15, filesize=11.6 K 2023-03-31 16:54:14,559 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/C/308dfc7bca3948e690cefe3994e99656 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/308dfc7bca3948e690cefe3994e99656 2023-03-31 16:54:14,563 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/308dfc7bca3948e690cefe3994e99656, entries=150, sequenceid=15, filesize=11.6 K 2023-03-31 16:54:14,564 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for e76385ddbd37af190ba2df92b5602e3b in 571ms, sequenceid=15, compaction requested=false 2023-03-31 16:54:14,565 DEBUG [MemStoreFlusher.0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-03-31 16:54:14,567 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e76385ddbd37af190ba2df92b5602e3b: 2023-03-31 16:54:14,604 DEBUG [Thread-907] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:14,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:14,607 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:14,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:14,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:14,632 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:14,632 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:14,632 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:14,632 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:14,633 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:14,633 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:14,635 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,635 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:14,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:14,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:14,635 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,635 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:14,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:14,636 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:14,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:14,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:14,636 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:14,645 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:14,645 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:14,645 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:14,645 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:14,645 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:14,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on e76385ddbd37af190ba2df92b5602e3b 2023-03-31 16:54:14,651 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e76385ddbd37af190ba2df92b5602e3b 3/3 column families, dataSize=140.89 KB heapSize=390.84 KB 2023-03-31 16:54:14,651 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e76385ddbd37af190ba2df92b5602e3b, store=A 2023-03-31 16:54:14,651 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:54:14,651 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e76385ddbd37af190ba2df92b5602e3b, store=B 2023-03-31 16:54:14,651 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:54:14,651 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e76385ddbd37af190ba2df92b5602e3b, store=C 2023-03-31 16:54:14,651 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:54:14,656 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:14,656 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:14,656 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:14,656 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. started... 2023-03-31 16:54:14,656 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. as already flushing 2023-03-31 16:54:14,656 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:14,656 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:14,656 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:14,656 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,662 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:14,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52906 deadline: 1680281714662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,664 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:14,664 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:14,669 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:14,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52872 deadline: 1680281714668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,669 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:14,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52920 deadline: 1680281714668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,670 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:14,670 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52878 deadline: 1680281714669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:14,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:14,670 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:14,670 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:14,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52892 deadline: 1680281714669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:14,671 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:14,672 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:14,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:14,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:14,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:14,674 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:14,675 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c58bb6a[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:14,675 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:14,675 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:14,677 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:14,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:14,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:14,677 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:14,677 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:14,677 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:14,681 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:14,681 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:14,681 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,681 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:14,681 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:14,682 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:14,682 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:14,682 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:14,682 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:14,682 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:14,682 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:14,683 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:14,683 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:14,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:14,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:14,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:14,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:14,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:14,683 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:14,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:14,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:14,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:14,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:14,686 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:14,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:14,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:14,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,688 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:14,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:14,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,692 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,692 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:14,692 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:14,692 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:14,692 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,692 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:14,692 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:14,692 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:14,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:14,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:14,693 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:14,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:14,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:14,694 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:14,694 DEBUG [Thread-907] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:14,694 DEBUG [Thread-907] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:14,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:14,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:14,749 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303312a8657c7e2be4bd7a957b94e55ecf000 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303312a8657c7e2be4bd7a957b94e55ecf000 2023-03-31 16:54:14,764 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:14,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52906 deadline: 1680281714764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,794 DEBUG [Thread-907] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:14,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:14,797 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:14,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:14,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:14,805 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:14,805 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:14,806 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:14,806 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:14,806 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:14,806 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:14,807 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,807 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:14,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:14,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:14,808 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,808 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:14,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:14,808 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:14,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:14,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:14,809 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:14,812 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:14,812 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:14,812 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:14,812 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:14,812 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:14,816 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:14,816 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:14,816 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:14,816 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. started... 2023-03-31 16:54:14,816 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. as already flushing 2023-03-31 16:54:14,816 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:14,816 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:14,816 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:14,816 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,816 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:14,816 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:14,818 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:14,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:14,818 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:14,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:14,818 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:14,819 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:14,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:14,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:14,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:14,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:14,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6fcb0e6b[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:14,820 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:14,820 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:14,823 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:14,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:14,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:14,823 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:14,823 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:14,823 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:14,823 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,823 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:14,824 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:14,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:14,824 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:14,825 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:14,825 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:14,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:14,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:14,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:14,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:14,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:14,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:14,827 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:14,827 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:14,828 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:14,828 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:14,828 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:14,828 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:14,830 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:14,830 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:14,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:14,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:14,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:14,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:14,830 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:14,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:14,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:14,830 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:14,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:14,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:14,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:14,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:14,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:14,832 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:14,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:14,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,834 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,834 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:14,834 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:14,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:14,834 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:14,834 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:14,834 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:14,834 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:14,834 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,834 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:14,834 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:14,834 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:14,834 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:14,834 DEBUG [Thread-907] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:14,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:14,834 DEBUG [Thread-907] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:14,835 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:14,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:14,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:14,935 DEBUG [Thread-907] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:14,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:14,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:14,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:14,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:14,944 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:14,944 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:14,944 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:14,944 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:14,945 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:14,945 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:14,946 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:14,946 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:14,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:14,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:14,947 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,947 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:14,947 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:14,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:14,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:14,947 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:14,948 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:14,948 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:14,948 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:14,948 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:14,948 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:14,952 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:14,952 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:14,952 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:14,952 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. started... 2023-03-31 16:54:14,952 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. as already flushing 2023-03-31 16:54:14,952 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:14,952 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:14,952 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:14,952 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,954 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:14,954 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:14,955 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:14,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:14,956 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:14,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:14,956 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:14,956 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:14,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:14,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:14,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:14,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:14,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2306cb52[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:14,958 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:14,958 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:14,959 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:14,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:14,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:14,959 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:14,959 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:14,959 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:14,959 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,959 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:14,964 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:14,964 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,964 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:14,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:14,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:14,965 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:14,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:14,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:14,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:14,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,967 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:14,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52906 deadline: 1680281714966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:14,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:14,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:14,968 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:14,968 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:14,968 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:14,968 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:14,969 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:14,969 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:14,970 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:14,970 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:14,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:14,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:14,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:14,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:14,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:14,970 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:14,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:14,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:14,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:14,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:14,972 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:14,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:14,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:14,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,973 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:14,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:14,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,977 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:14,977 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:14,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:14,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:14,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:14,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:14,977 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:14,977 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,977 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:14,977 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:14,977 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:14,977 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:14,977 DEBUG [Thread-907] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:14,977 DEBUG [Thread-907] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:14,977 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:14,978 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:14,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:14,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:15,078 DEBUG [Thread-907] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:15,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:15,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:15,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:15,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:15,082 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:15,082 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:15,082 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:15,082 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:15,082 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:15,082 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:15,083 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:15,083 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:15,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:15,084 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,084 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:15,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:15,084 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:15,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:15,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:15,085 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:15,085 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:15,085 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:15,085 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:15,085 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:15,085 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:15,086 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:15,086 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:15,086 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:15,086 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. started... 2023-03-31 16:54:15,086 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. as already flushing 2023-03-31 16:54:15,086 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:15,086 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:15,086 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:15,086 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,086 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:15,086 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:15,089 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,089 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:15,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:15,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:15,089 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:15,089 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:15,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:15,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:15,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:15,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:15,091 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:15,091 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5dc11433[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:15,091 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:15,091 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:15,092 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:15,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:15,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:15,092 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:15,092 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:15,092 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:15,092 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,092 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:15,093 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,093 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:15,094 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:15,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:15,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:15,094 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:15,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:15,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:15,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:15,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:15,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:15,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:15,096 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:15,096 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:15,097 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:15,097 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:15,097 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:15,097 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:15,098 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:15,098 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:15,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:15,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:15,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:15,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:15,098 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:15,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:15,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:15,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:15,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:15,099 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:15,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:15,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:15,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:15,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,100 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:15,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:15,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,104 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,104 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:15,104 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:15,104 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:15,104 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:15,104 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:15,104 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:15,104 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,104 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:15,104 DEBUG [Thread-907] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:15,104 DEBUG [Thread-907] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:15,104 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:15,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:15,104 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:15,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:15,104 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:15,105 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:15,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:15,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:15,171 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=50, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/A/caf1fcac7bcf41e99bede9c8d7470dfb 2023-03-31 16:54:15,174 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:15,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52872 deadline: 1680281715173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,176 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:15,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52892 deadline: 1680281715175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,178 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:15,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52878 deadline: 1680281715178, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,179 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:15,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52920 deadline: 1680281715178, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,204 DEBUG [Thread-907] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:15,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:15,207 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:15,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:15,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:15,212 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:15,212 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:15,212 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:15,212 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:15,213 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:15,213 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:15,214 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:15,214 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:15,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:15,214 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,215 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:15,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:15,215 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:15,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:15,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:15,216 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:15,223 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:15,223 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:15,223 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:15,223 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:15,223 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:15,224 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:15,224 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:15,224 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:15,224 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. started... 2023-03-31 16:54:15,224 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. as already flushing 2023-03-31 16:54:15,224 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:15,224 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:15,224 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:15,224 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,224 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:15,224 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:15,226 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,226 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:15,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:15,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:15,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:15,227 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:15,227 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:15,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:15,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:15,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:15,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:15,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54cb770a[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:15,228 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:15,228 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:15,230 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:15,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:15,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:15,230 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:15,230 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:15,230 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:15,237 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:15,237 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,237 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:15,237 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:15,237 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:15,237 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:15,237 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:15,237 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:15,238 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:15,238 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:15,238 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:15,239 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:15,239 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:15,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:15,239 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:15,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:15,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:15,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:15,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:15,240 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:15,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:15,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:15,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:15,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:15,241 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:15,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:15,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:15,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:15,243 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,243 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:15,243 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:15,244 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:15,244 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:15,244 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,244 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:15,244 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:15,244 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:15,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:15,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:15,244 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:15,244 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:15,244 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:15,245 DEBUG [Thread-907] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:15,245 DEBUG [Thread-907] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:15,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:15,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:15,245 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:15,269 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:15,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52906 deadline: 1680281715268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,345 DEBUG [Thread-907] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:15,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:15,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:15,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:15,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:15,354 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:15,354 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:15,354 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:15,354 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:15,355 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:15,355 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:15,359 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:15,359 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:15,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:15,359 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,359 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:15,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:15,360 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:15,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:15,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:15,360 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:15,360 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:15,360 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:15,361 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:15,361 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:15,361 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:15,362 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:15,362 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:15,362 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:15,362 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. started... 2023-03-31 16:54:15,362 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. as already flushing 2023-03-31 16:54:15,362 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:15,362 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:15,362 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:15,362 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,362 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:15,362 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:15,364 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,364 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:15,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:15,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:15,364 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:15,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:15,364 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:15,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:15,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:15,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:15,373 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:15,373 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@634483ae[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:15,373 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:15,373 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:15,374 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:15,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:15,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:15,374 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,374 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:15,374 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:15,374 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:15,374 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:15,376 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,376 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:15,376 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:15,376 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:15,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:15,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:15,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:15,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:15,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:15,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:15,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:15,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:15,378 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:15,378 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:15,378 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:15,379 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:15,379 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:15,379 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:15,383 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:15,383 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:15,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:15,383 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:15,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:15,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:15,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:15,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:15,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:15,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:15,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:15,384 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:15,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:15,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:15,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:15,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,386 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:15,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:15,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,390 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,390 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:15,390 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:15,390 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,390 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:15,390 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:15,390 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:15,390 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:15,391 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:15,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:15,390 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:15,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:15,391 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:15,391 DEBUG [Thread-907] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:15,391 DEBUG [Thread-907] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:15,391 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:15,391 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:15,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:15,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:15,491 DEBUG [Thread-907] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:15,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:15,495 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:15,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:15,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:15,497 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:15,497 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:15,497 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:15,497 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:15,497 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:15,497 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:15,499 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,500 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:15,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:15,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:15,500 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,500 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:15,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:15,501 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:15,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:15,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:15,501 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:15,502 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:15,502 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:15,502 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:15,502 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:15,502 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:15,502 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:15,502 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:15,502 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:15,502 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. started... 2023-03-31 16:54:15,502 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. as already flushing 2023-03-31 16:54:15,502 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:15,502 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:15,502 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:15,502 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,503 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:15,503 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:15,504 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,504 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:15,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:15,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:15,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:15,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:15,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:15,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,506 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:15,506 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:15,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:15,506 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:15,507 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@768d98cf[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:15,507 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:15,507 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:15,508 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:15,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:15,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:15,508 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:15,508 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:15,508 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:15,508 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,508 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:15,509 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:15,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:15,510 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:15,510 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:15,510 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:15,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:15,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:15,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:15,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:15,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:15,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:15,512 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:15,512 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:15,513 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:15,513 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:15,513 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:15,513 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:15,517 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:15,517 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:15,517 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:15,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:15,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:15,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:15,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:15,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:15,518 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:15,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:15,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:15,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:15,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:15,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:15,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:15,520 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:15,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:15,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,522 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,522 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:15,522 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:15,522 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,522 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:15,522 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:15,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:15,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:15,523 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:15,523 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:15,523 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:15,523 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:15,523 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:15,523 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:15,523 DEBUG [Thread-907] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:15,523 DEBUG [Thread-907] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:15,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:15,523 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:15,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:15,593 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/B/9618f2346f0e4219afe0af8176a26ca9 2023-03-31 16:54:15,623 DEBUG [Thread-907] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:15,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:15,626 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:15,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:15,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:15,627 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:15,627 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:15,627 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:15,627 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:15,628 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:15,628 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:15,630 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:15,630 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:15,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:15,630 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,630 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:15,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:15,630 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:15,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:15,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:15,631 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:15,631 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:15,631 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:15,631 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:15,631 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:15,632 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:15,632 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:15,632 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:15,632 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:15,632 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. started... 2023-03-31 16:54:15,632 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. as already flushing 2023-03-31 16:54:15,632 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:15,632 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:15,632 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:15,632 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,632 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:15,632 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:15,634 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,634 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:15,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:15,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:15,634 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:15,634 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:15,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:15,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:15,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:15,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:15,636 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:15,636 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23b87aa1[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:15,636 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:15,636 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:15,637 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,637 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:15,639 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:15,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:15,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:15,639 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:15,639 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:15,639 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:15,642 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:15,642 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,642 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:15,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:15,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:15,642 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:15,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:15,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:15,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:15,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:15,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:15,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:15,646 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:15,646 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:15,646 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:15,646 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:15,646 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:15,646 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:15,648 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:15,648 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:15,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:15,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:15,649 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:15,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:15,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:15,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:15,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:15,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:15,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:15,649 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:15,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:15,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:15,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:15,650 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:15,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:15,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,653 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,653 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:15,653 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:15,653 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:15,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:15,653 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:15,653 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:15,653 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:15,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:15,653 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:15,653 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,653 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:15,653 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:15,653 DEBUG [Thread-907] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:15,653 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:15,653 DEBUG [Thread-907] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:15,654 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:15,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:15,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:15,723 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-31 16:54:15,753 DEBUG [Thread-907] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:15,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:15,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:15,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:15,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:15,758 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:15,758 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:15,758 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:15,758 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:15,758 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:15,758 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:15,759 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,759 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:15,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:15,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:15,760 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,760 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:15,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:15,760 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:15,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:15,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:15,761 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:15,761 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:15,761 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:15,761 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:15,761 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:15,761 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:15,768 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:15,768 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:15,768 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:15,768 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. started... 2023-03-31 16:54:15,768 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. as already flushing 2023-03-31 16:54:15,768 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:15,768 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:15,768 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:15,768 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,768 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:15,768 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:15,771 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:15,771 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:15,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:15,771 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:15,771 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:15,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:15,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:15,772 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:15,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52906 deadline: 1680281715770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:15,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:15,776 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:15,776 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c442afb[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:15,776 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:15,776 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:15,777 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:15,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:15,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:15,777 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:15,777 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:15,777 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:15,778 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,778 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:15,779 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:15,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:15,779 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:15,779 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:15,779 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:15,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:15,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:15,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:15,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:15,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:15,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:15,785 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:15,786 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:15,786 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:15,786 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:15,786 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:15,786 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:15,788 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:15,788 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:15,788 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:15,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:15,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:15,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:15,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:15,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:15,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:15,789 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:15,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:15,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:15,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:15,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:15,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:15,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,790 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:15,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:15,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,793 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,793 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:15,793 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:15,793 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:15,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:15,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:15,793 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:15,793 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:15,794 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:15,793 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:15,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:15,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:15,794 DEBUG [Thread-907] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:15,794 DEBUG [Thread-907] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:15,794 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,794 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:15,794 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:15,794 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:15,894 DEBUG [Thread-907] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:15,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:15,897 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:15,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:15,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:15,898 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:15,898 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:15,898 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:15,898 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:15,898 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:15,899 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:15,900 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:15,900 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:15,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:15,900 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,900 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:15,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:15,901 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:15,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:15,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:15,901 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:15,907 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:15,907 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:15,907 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:15,907 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:15,907 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:15,907 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:15,907 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:15,907 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:15,907 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. started... 2023-03-31 16:54:15,907 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. as already flushing 2023-03-31 16:54:15,907 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:15,907 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:15,907 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:15,908 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,908 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:15,908 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:15,909 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,909 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:15,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:15,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:15,909 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:15,910 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:15,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:15,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:15,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:15,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:15,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:15,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6bda952e[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:15,912 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:15,912 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:15,913 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:15,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:15,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:15,914 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:15,914 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,914 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:15,914 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:15,914 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:15,917 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:15,917 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:15,917 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:15,917 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:15,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:15,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:15,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:15,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:15,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:15,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:15,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:15,920 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:15,920 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:15,920 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:15,920 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:15,920 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:15,920 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:15,923 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:15,923 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:15,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:15,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:15,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:15,923 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:15,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:15,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:15,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:15,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:15,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:15,924 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:15,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:15,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:15,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:15,926 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,928 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:15,928 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:15,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:15,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:15,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:15,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:15,930 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,930 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:15,930 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:15,930 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:15,930 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:15,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:15,930 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:15,930 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:15,930 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:15,930 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:15,930 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:15,930 DEBUG [Thread-907] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:15,930 DEBUG [Thread-907] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:15,934 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:16,006 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/C/1582f5e7f9d246fa8594e5860f39e221 2023-03-31 16:54:16,011 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/A/caf1fcac7bcf41e99bede9c8d7470dfb as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/caf1fcac7bcf41e99bede9c8d7470dfb 2023-03-31 16:54:16,016 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/caf1fcac7bcf41e99bede9c8d7470dfb, entries=200, sequenceid=50, filesize=31.9 K 2023-03-31 16:54:16,017 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/B/9618f2346f0e4219afe0af8176a26ca9 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/9618f2346f0e4219afe0af8176a26ca9 2023-03-31 16:54:16,021 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/9618f2346f0e4219afe0af8176a26ca9, entries=150, sequenceid=50, filesize=11.6 K 2023-03-31 16:54:16,022 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/C/1582f5e7f9d246fa8594e5860f39e221 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/1582f5e7f9d246fa8594e5860f39e221 2023-03-31 16:54:16,027 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/1582f5e7f9d246fa8594e5860f39e221, entries=150, sequenceid=50, filesize=11.6 K 2023-03-31 16:54:16,028 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~390.80 KB/400176, currentSize=46.96 KB/48090 for e76385ddbd37af190ba2df92b5602e3b in 1377ms, sequenceid=50, compaction requested=false 2023-03-31 16:54:16,028 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e76385ddbd37af190ba2df92b5602e3b: 2023-03-31 16:54:16,031 DEBUG [Thread-907] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:16,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:16,033 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:16,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:16,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:16,034 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:16,034 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:16,034 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:16,035 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:16,035 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:16,035 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:16,036 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:16,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:16,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:16,037 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:16,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:16,037 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:16,037 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:16,037 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:16,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:16,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:16,038 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:16,040 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:16,040 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:16,040 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:16,040 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:16,040 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:16,048 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:16,048 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:16,048 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:16,048 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. started... 2023-03-31 16:54:16,048 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e76385ddbd37af190ba2df92b5602e3b 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-31 16:54:16,049 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e76385ddbd37af190ba2df92b5602e3b, store=A 2023-03-31 16:54:16,049 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:16,049 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e76385ddbd37af190ba2df92b5602e3b, store=B 2023-03-31 16:54:16,049 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:16,049 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e76385ddbd37af190ba2df92b5602e3b, store=C 2023-03-31 16:54:16,049 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:16,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on e76385ddbd37af190ba2df92b5602e3b 2023-03-31 16:54:16,182 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. as already flushing 2023-03-31 16:54:16,208 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680281656193, max timestamp=1680281656205Num uniques 859; 2023-03-31 16:54:16,208 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680281656193, max timestamp=1680281656205Num uniques 859; 2023-03-31 16:54:16,208 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-31 16:54:16,208 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-31 16:54:16,208 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680281656193, max timestamp=1680281656205Num uniques 859; 2023-03-31 16:54:16,208 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-31 16:54:16,231 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:16,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52878 deadline: 1680281716229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:16,232 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:16,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52920 deadline: 1680281716229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:16,233 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:16,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52872 deadline: 1680281716230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:16,238 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:16,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52892 deadline: 1680281716236, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:16,333 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:16,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52920 deadline: 1680281716333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:16,334 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:16,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52872 deadline: 1680281716334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:16,338 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:16,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52878 deadline: 1680281716337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:16,343 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:16,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52892 deadline: 1680281716340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:16,483 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303314d7540a0dafa4cf59c4e0633a0d9ed62 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303314d7540a0dafa4cf59c4e0633a0d9ed62 2023-03-31 16:54:16,539 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:16,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52920 deadline: 1680281716536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:16,539 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:16,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52872 deadline: 1680281716536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:16,545 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:16,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52878 deadline: 1680281716541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:16,546 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:16,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52892 deadline: 1680281716545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:16,774 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:16,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52906 deadline: 1680281716774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:16,842 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:16,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52920 deadline: 1680281716842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:16,842 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:16,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52872 deadline: 1680281716842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:16,847 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:16,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52878 deadline: 1680281716846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:16,850 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:16,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52892 deadline: 1680281716848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:16,890 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=60, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/A/7125aeff4e0e423282a4bf05dc0fd495 2023-03-31 16:54:16,923 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=60 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/B/8ec4b187750e43ada713cc6e720ee32c 2023-03-31 16:54:16,947 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=60 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/C/f7265421edeb4afab79c0e61a58898ca 2023-03-31 16:54:16,954 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/A/7125aeff4e0e423282a4bf05dc0fd495 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/7125aeff4e0e423282a4bf05dc0fd495 2023-03-31 16:54:16,960 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/7125aeff4e0e423282a4bf05dc0fd495, entries=150, sequenceid=60, filesize=25.1 K 2023-03-31 16:54:16,961 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/B/8ec4b187750e43ada713cc6e720ee32c as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/8ec4b187750e43ada713cc6e720ee32c 2023-03-31 16:54:16,965 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/8ec4b187750e43ada713cc6e720ee32c, entries=150, sequenceid=60, filesize=11.6 K 2023-03-31 16:54:16,966 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/C/f7265421edeb4afab79c0e61a58898ca as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/f7265421edeb4afab79c0e61a58898ca 2023-03-31 16:54:16,971 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/f7265421edeb4afab79c0e61a58898ca, entries=150, sequenceid=60, filesize=11.6 K 2023-03-31 16:54:16,974 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for e76385ddbd37af190ba2df92b5602e3b in 926ms, sequenceid=60, compaction requested=true 2023-03-31 16:54:16,974 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e76385ddbd37af190ba2df92b5602e3b: 2023-03-31 16:54:16,974 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:16,974 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:16,974 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:16,974 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:16,974 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:16,974 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:16,976 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:16,976 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:16,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:16,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:16,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:16,976 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:16,976 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:16,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:16,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:16,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:16,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:16,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:16,979 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:16,979 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@718b337a[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:16,979 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:16,979 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:16,980 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:16,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:16,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:16,980 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:16,981 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:16,981 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:16,981 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:16,981 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:16,984 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:16,984 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:16,984 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:16,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:16,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:16,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:16,984 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:16,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:16,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:16,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:16,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:16,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:16,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:16,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:16,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:16,988 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:16,988 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:16,988 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:16,988 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:16,988 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:16,989 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:16,991 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:16,991 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:16,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:16,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:16,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:16,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:16,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:16,991 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:16,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:16,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:16,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:16,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:16,992 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:16,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:16,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:16,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:16,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:16,994 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:16,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:16,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:16,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:16,996 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:16,996 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:16,996 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:16,996 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:16,996 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:16,996 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:16,996 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:16,997 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:16,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:16,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:16,996 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:16,997 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:16,997 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:16,997 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:16,997 DEBUG [Thread-907] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:16,997 DEBUG [Thread-907] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:16,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:16,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:17,097 DEBUG [Thread-907] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:17,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:17,100 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:17,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:17,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:17,101 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:17,101 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:17,101 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:17,101 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:17,102 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:17,102 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:17,103 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:17,103 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:17,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:17,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:17,103 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:17,103 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:17,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:17,104 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:17,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:17,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:17,105 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:17,105 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:17,105 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:17,106 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:17,106 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:17,106 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:17,106 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:17,106 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:17,106 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:17,106 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. started... 2023-03-31 16:54:17,106 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e76385ddbd37af190ba2df92b5602e3b 3/3 column families, dataSize=140.89 KB heapSize=390.84 KB 2023-03-31 16:54:17,106 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e76385ddbd37af190ba2df92b5602e3b, store=A 2023-03-31 16:54:17,107 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:54:17,107 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e76385ddbd37af190ba2df92b5602e3b, store=B 2023-03-31 16:54:17,107 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:54:17,107 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e76385ddbd37af190ba2df92b5602e3b, store=C 2023-03-31 16:54:17,107 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:54:17,129 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230331c5139450d6e2450eb9b2506c301e836f to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331c5139450d6e2450eb9b2506c301e836f 2023-03-31 16:54:17,146 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=95, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/A/30aa652862a947dfbc88722a0caa6743 2023-03-31 16:54:17,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on e76385ddbd37af190ba2df92b5602e3b 2023-03-31 16:54:17,350 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. as already flushing 2023-03-31 16:54:17,362 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:17,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52878 deadline: 1680281717355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:17,362 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:17,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52872 deadline: 1680281717355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:17,365 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:17,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52920 deadline: 1680281717361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:17,365 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:17,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52892 deadline: 1680281717362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:17,465 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:17,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52878 deadline: 1680281717463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:17,465 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:17,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52872 deadline: 1680281717463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:17,468 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:17,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52920 deadline: 1680281717466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:17,468 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:17,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52892 deadline: 1680281717466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:17,562 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=95 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/B/cffe11007d554e74ba6f7b376f31e142 2023-03-31 16:54:17,668 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:17,668 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:17,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52878 deadline: 1680281717666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:17,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52872 deadline: 1680281717667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:17,671 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:17,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52920 deadline: 1680281717669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:17,672 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:17,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52892 deadline: 1680281717671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:17,971 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:17,971 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:17,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52878 deadline: 1680281717970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:17,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52872 deadline: 1680281717970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:17,974 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=95 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/C/0172887afb7a44b2bf66317ee3b84e47 2023-03-31 16:54:17,976 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:17,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52892 deadline: 1680281717973, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:17,977 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:17,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52920 deadline: 1680281717974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:17,979 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/A/30aa652862a947dfbc88722a0caa6743 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/30aa652862a947dfbc88722a0caa6743 2023-03-31 16:54:17,983 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/30aa652862a947dfbc88722a0caa6743, entries=150, sequenceid=95, filesize=25.1 K 2023-03-31 16:54:17,984 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/B/cffe11007d554e74ba6f7b376f31e142 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/cffe11007d554e74ba6f7b376f31e142 2023-03-31 16:54:17,989 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/cffe11007d554e74ba6f7b376f31e142, entries=150, sequenceid=95, filesize=11.6 K 2023-03-31 16:54:17,990 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/C/0172887afb7a44b2bf66317ee3b84e47 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/0172887afb7a44b2bf66317ee3b84e47 2023-03-31 16:54:17,996 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/0172887afb7a44b2bf66317ee3b84e47, entries=150, sequenceid=95, filesize=11.6 K 2023-03-31 16:54:17,996 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~390.80 KB/400176, currentSize=46.96 KB/48090 for e76385ddbd37af190ba2df92b5602e3b in 890ms, sequenceid=95, compaction requested=true 2023-03-31 16:54:17,997 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e76385ddbd37af190ba2df92b5602e3b: 2023-03-31 16:54:17,997 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:17,997 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:17,997 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:17,997 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:17,997 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:17,997 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:17,998 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:17,998 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:17,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:17,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:17,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:17,999 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:17,999 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:17,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:17,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:17,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:18,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:18,000 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:18,000 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11062298[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:18,000 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:18,000 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:18,003 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:18,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:18,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:18,003 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:18,003 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:18,003 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:18,003 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,003 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:18,004 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,005 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:18,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:18,005 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:18,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:18,005 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:18,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:18,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:18,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:18,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:18,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:18,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:18,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:18,007 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:18,007 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:18,007 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:18,007 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:18,007 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:18,009 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:18,009 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:18,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:18,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:18,009 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:18,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:18,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:18,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:18,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:18,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:18,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:18,010 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:18,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:18,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:18,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:18,012 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:18,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:18,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,018 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,018 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:18,018 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:18,018 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:18,018 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:18,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:18,019 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:18,019 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,019 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:18,019 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:18,019 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:18,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:18,019 DEBUG [Thread-907] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:18,018 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:18,019 DEBUG [Thread-907] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:18,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:18,019 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:18,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:18,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:18,119 DEBUG [Thread-907] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:18,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:18,121 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:18,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:18,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:18,127 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:18,127 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:18,127 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:18,127 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:18,127 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:18,127 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:18,128 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:18,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:18,129 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:18,129 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,129 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:18,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:18,129 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:18,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:18,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:18,130 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:18,130 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:18,130 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:18,130 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:18,130 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:18,130 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:18,130 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:18,130 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:18,130 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:18,130 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. started... 2023-03-31 16:54:18,130 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e76385ddbd37af190ba2df92b5602e3b 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-31 16:54:18,131 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e76385ddbd37af190ba2df92b5602e3b, store=A 2023-03-31 16:54:18,131 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:18,131 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e76385ddbd37af190ba2df92b5602e3b, store=B 2023-03-31 16:54:18,131 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:18,131 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e76385ddbd37af190ba2df92b5602e3b, store=C 2023-03-31 16:54:18,131 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:18,147 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230331819a5a6dec8849e49b1e88504f946199 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331819a5a6dec8849e49b1e88504f946199 2023-03-31 16:54:18,153 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=105, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/A/9ae7627fbf8b4536a2a289cfae4492ed 2023-03-31 16:54:18,179 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=105 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/B/b87470db93dc46e7b06273abafeb16b9 2023-03-31 16:54:18,200 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=105 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/C/329c493d8e7f4c79aa85852a3e5dcdd1 2023-03-31 16:54:18,205 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/A/9ae7627fbf8b4536a2a289cfae4492ed as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/9ae7627fbf8b4536a2a289cfae4492ed 2023-03-31 16:54:18,209 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/9ae7627fbf8b4536a2a289cfae4492ed, entries=150, sequenceid=105, filesize=25.1 K 2023-03-31 16:54:18,210 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/B/b87470db93dc46e7b06273abafeb16b9 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/b87470db93dc46e7b06273abafeb16b9 2023-03-31 16:54:18,216 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/b87470db93dc46e7b06273abafeb16b9, entries=150, sequenceid=105, filesize=11.6 K 2023-03-31 16:54:18,217 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/C/329c493d8e7f4c79aa85852a3e5dcdd1 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/329c493d8e7f4c79aa85852a3e5dcdd1 2023-03-31 16:54:18,229 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/329c493d8e7f4c79aa85852a3e5dcdd1, entries=150, sequenceid=105, filesize=11.6 K 2023-03-31 16:54:18,230 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=0 B/0 for e76385ddbd37af190ba2df92b5602e3b in 100ms, sequenceid=105, compaction requested=true 2023-03-31 16:54:18,230 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e76385ddbd37af190ba2df92b5602e3b: 2023-03-31 16:54:18,230 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:18,231 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:18,231 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:18,231 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,231 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:18,231 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:18,233 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,233 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:18,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:18,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:18,233 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:18,233 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:18,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:18,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:18,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:18,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:18,235 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:18,235 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c98d1b1[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:18,235 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:18,235 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:18,236 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:18,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:18,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:18,237 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:18,237 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:18,237 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:18,237 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,237 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:18,239 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:18,239 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:18,239 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:18,239 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:18,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:18,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:18,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:18,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:18,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:18,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:18,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:18,242 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:18,242 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:18,243 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:18,243 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:18,243 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:18,243 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:18,244 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:18,244 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:18,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:18,245 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:18,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:18,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:18,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:18,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:18,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:18,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:18,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:18,245 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:18,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:18,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:18,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:18,247 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:18,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:18,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,250 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:18,250 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:18,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:18,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:18,250 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,250 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:18,250 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:18,250 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:18,250 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:18,250 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:18,250 DEBUG [Thread-907] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:18,250 DEBUG [Thread-907] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:18,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:18,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:18,250 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:18,251 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,251 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:18,251 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:18,251 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:18,351 DEBUG [Thread-907] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:18,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:18,353 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:18,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:18,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:18,355 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:18,355 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:18,355 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:18,355 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:18,355 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:18,355 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:18,357 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,357 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:18,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:18,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:18,357 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,357 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:18,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:18,358 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:18,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:18,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:18,358 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:18,359 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:18,359 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:18,359 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:18,359 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:18,359 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:18,360 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:18,360 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:18,360 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:18,360 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. started... 2023-03-31 16:54:18,360 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e76385ddbd37af190ba2df92b5602e3b: 2023-03-31 16:54:18,360 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:18,361 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:18,361 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:18,361 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,361 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:18,361 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:18,363 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,363 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:18,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:18,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:18,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:18,364 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:18,364 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:18,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:18,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:18,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:18,366 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:18,366 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:18,366 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@235444e9[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:18,366 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:18,368 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:18,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:18,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:18,368 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:18,368 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:18,368 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:18,368 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,368 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:18,370 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,370 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:18,370 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:18,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:18,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:18,370 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:18,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:18,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:18,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:18,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:18,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:18,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:18,373 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:18,373 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:18,373 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:18,373 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:18,373 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:18,373 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:18,375 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:18,375 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:18,375 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:18,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:18,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:18,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:18,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:18,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:18,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:18,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:18,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:18,376 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:18,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:18,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:18,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:18,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,377 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:18,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:18,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,381 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:18,381 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:18,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:18,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:18,381 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,381 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:18,381 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:18,381 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:18,381 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:18,381 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:18,381 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,381 DEBUG [Thread-907] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:18,381 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:18,381 DEBUG [Thread-907] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:18,382 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:18,382 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:18,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:18,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:18,482 DEBUG [Thread-907] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:18,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:18,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:18,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:18,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:18,486 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:18,486 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:18,486 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:18,486 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:18,486 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:18,486 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:18,488 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,488 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:18,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:18,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:18,489 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,489 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:18,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:18,489 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:18,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:18,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:18,490 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:18,490 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:18,490 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:18,490 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:18,490 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:18,490 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:18,491 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:18,491 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:18,491 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:18,491 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. started... 2023-03-31 16:54:18,492 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e76385ddbd37af190ba2df92b5602e3b: 2023-03-31 16:54:18,492 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:18,492 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:18,492 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:18,492 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,492 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:18,492 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:18,493 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,493 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:18,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:18,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:18,494 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:18,494 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:18,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:18,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:18,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:18,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:18,496 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:18,496 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b6ca31a[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:18,496 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:18,496 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:18,501 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:18,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:18,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:18,501 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:18,501 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,501 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:18,501 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:18,502 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:18,504 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:18,504 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,504 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:18,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:18,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:18,504 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:18,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:18,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:18,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:18,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:18,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:18,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:18,508 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:18,508 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:18,508 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:18,508 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:18,508 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:18,508 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:18,510 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:18,511 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:18,511 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:18,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:18,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:18,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:18,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:18,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:18,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:18,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:18,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:18,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:18,512 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:18,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:18,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:18,514 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:18,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:18,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,516 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:18,516 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:18,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:18,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:18,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:18,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:18,516 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:18,516 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,517 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:18,517 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:18,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:18,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:18,517 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,517 DEBUG [Thread-907] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:18,517 DEBUG [Thread-907] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:18,517 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:18,517 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:18,517 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:18,517 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:18,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on e76385ddbd37af190ba2df92b5602e3b 2023-03-31 16:54:18,554 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e76385ddbd37af190ba2df92b5602e3b 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-31 16:54:18,555 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e76385ddbd37af190ba2df92b5602e3b, store=A 2023-03-31 16:54:18,555 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:18,555 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e76385ddbd37af190ba2df92b5602e3b, store=B 2023-03-31 16:54:18,555 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:18,555 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e76385ddbd37af190ba2df92b5602e3b, store=C 2023-03-31 16:54:18,555 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:18,572 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023033114b0636d959a40f9b6ad97e7453d89c9 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033114b0636d959a40f9b6ad97e7453d89c9 2023-03-31 16:54:18,589 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1680281658558, max timestamp=1680281658588Num uniques 859; 2023-03-31 16:54:18,589 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1680281658558, max timestamp=1680281658588Num uniques 859; 2023-03-31 16:54:18,589 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-03-31 16:54:18,589 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-03-31 16:54:18,589 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1680281658558, max timestamp=1680281658588Num uniques 859; 2023-03-31 16:54:18,589 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-03-31 16:54:18,617 DEBUG [Thread-907] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:18,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:18,620 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:18,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:18,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:18,621 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:18,621 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:18,621 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:18,621 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:18,621 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:18,621 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:18,622 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:18,622 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:18,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:18,623 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,623 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:18,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:18,623 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:18,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:18,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:18,624 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:18,624 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:18,624 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:18,624 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:18,624 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:18,624 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:18,624 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:18,624 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:18,624 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:18,624 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. started... 2023-03-31 16:54:18,624 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. as already flushing 2023-03-31 16:54:18,624 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:18,624 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:18,624 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:18,624 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,624 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:18,625 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:18,627 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:18,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52892 deadline: 1680281718590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,633 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,633 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:18,633 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:18,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:18,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:18,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52872 deadline: 1680281718627, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,634 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:18,634 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:18,634 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:18,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52920 deadline: 1680281718627, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:18,634 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:18,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52878 deadline: 1680281718628, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:18,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:18,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:18,636 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:18,636 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2bde420d[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:18,636 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:18,636 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:18,638 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:18,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:18,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:18,638 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:18,638 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:18,638 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:18,638 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,638 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:18,639 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:18,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:18,639 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:18,639 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:18,639 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:18,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:18,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:18,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:18,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:18,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:18,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:18,642 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:18,642 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:18,642 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:18,642 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:18,642 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:18,642 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:18,643 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:18,643 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:18,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:18,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:18,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:18,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:18,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:18,644 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:18,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:18,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:18,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:18,644 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:18,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:18,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:18,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:18,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,645 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:18,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:18,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,649 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:18,649 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:18,649 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:18,649 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:18,649 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:18,649 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,649 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:18,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:18,649 DEBUG [Thread-907] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:18,649 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:18,649 DEBUG [Thread-907] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:18,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:18,649 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:18,650 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,650 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:18,650 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:18,650 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:18,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:18,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:18,730 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:18,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52892 deadline: 1680281718728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,737 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:18,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52872 deadline: 1680281718735, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,737 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:18,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52920 deadline: 1680281718735, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,738 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:18,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52878 deadline: 1680281718735, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,750 DEBUG [Thread-907] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:18,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:18,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:18,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:18,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:18,753 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:18,753 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:18,753 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:18,753 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:18,754 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:18,754 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:18,755 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:18,755 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:18,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:18,755 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,755 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:18,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:18,756 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:18,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:18,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:18,756 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:18,756 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:18,756 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:18,757 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:18,757 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:18,757 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:18,757 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:18,757 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:18,757 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:18,757 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. started... 2023-03-31 16:54:18,757 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. as already flushing 2023-03-31 16:54:18,757 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:18,757 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:18,757 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:18,757 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,757 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:18,757 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:18,759 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,759 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:18,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:18,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:18,759 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:18,759 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:18,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:18,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:18,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:18,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:18,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:18,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71110b9e[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:18,761 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:18,761 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:18,763 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:18,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:18,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:18,764 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:18,764 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:18,764 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:18,764 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,764 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:18,765 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,765 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:18,765 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:18,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:18,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:18,765 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:18,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:18,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:18,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:18,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:18,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:18,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:18,768 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:18,768 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:18,768 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:18,768 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:18,768 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:18,768 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:18,770 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:18,770 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:18,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:18,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:18,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:18,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:18,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:18,770 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:18,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:18,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:18,771 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:18,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:18,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:18,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:18,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:18,772 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:18,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:18,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,774 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,774 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:18,774 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:18,774 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,774 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:18,774 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:18,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:18,775 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:18,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:18,775 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:18,775 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:18,775 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:18,775 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:18,775 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:18,775 DEBUG [Thread-907] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:18,775 DEBUG [Thread-907] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:18,775 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:18,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:18,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:18,793 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:18,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52906 deadline: 1680281718791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,795 DEBUG [Thread-905] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4132 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b., hostname=jenkins-hbase4.apache.org,33441,1680281619375, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-31 16:54:18,875 DEBUG [Thread-907] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:18,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:18,880 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:18,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:18,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:18,895 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:18,895 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:18,895 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:18,895 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:18,896 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:18,896 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:18,898 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,898 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:18,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:18,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:18,898 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,898 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:18,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:18,898 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:18,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:18,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:18,899 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:18,899 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:18,899 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:18,899 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:18,899 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:18,899 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:18,903 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:18,903 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:18,903 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:18,903 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. started... 2023-03-31 16:54:18,903 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. as already flushing 2023-03-31 16:54:18,903 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:18,903 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:18,903 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:18,903 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,903 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:18,903 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:18,905 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,905 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:18,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:18,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:18,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:18,906 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:18,906 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:18,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:18,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:18,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:18,907 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:18,907 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70636a59[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:18,907 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:18,908 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:18,909 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:18,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:18,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:18,910 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:18,910 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:18,910 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:18,910 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,910 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:18,911 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,911 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:18,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:18,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:18,911 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:18,912 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:18,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:18,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:18,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:18,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:18,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:18,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:18,916 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:18,917 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:18,917 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:18,917 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:18,917 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:18,917 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:18,919 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:18,919 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:18,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:18,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:18,919 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:18,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:18,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:18,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:18,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:18,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:18,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:18,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:18,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:18,921 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:18,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:18,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:18,923 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:18,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,927 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:18,928 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:18,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:18,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:18,928 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:18,928 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:18,928 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,928 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:18,928 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:18,928 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:18,928 DEBUG [Thread-907] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:18,928 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:18,928 DEBUG [Thread-907] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:18,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:18,928 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:18,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:18,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:18,938 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:18,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52892 deadline: 1680281718933, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,941 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:18,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52872 deadline: 1680281718938, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,942 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:18,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52878 deadline: 1680281718940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,942 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:18,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52920 deadline: 1680281718940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:18,978 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=118, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/A/a1beb4ab665c4d69a714f89128eaa846 2023-03-31 16:54:18,999 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=118 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/B/0fc1625fa8f74442bb50483deca7510c 2023-03-31 16:54:19,021 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=118 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/C/4e399327d7954581853ef1d4ce813b37 2023-03-31 16:54:19,028 DEBUG [Thread-907] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:19,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:19,042 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/A/a1beb4ab665c4d69a714f89128eaa846 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/a1beb4ab665c4d69a714f89128eaa846 2023-03-31 16:54:19,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:19,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:19,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:19,046 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:19,046 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:19,046 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:19,046 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:19,047 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:19,047 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:19,047 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/a1beb4ab665c4d69a714f89128eaa846, entries=400, sequenceid=118, filesize=59.0 K 2023-03-31 16:54:19,049 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/B/0fc1625fa8f74442bb50483deca7510c as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/0fc1625fa8f74442bb50483deca7510c 2023-03-31 16:54:19,049 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:19,049 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:19,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:19,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:19,050 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:19,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:19,050 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:19,050 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:19,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:19,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:19,051 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:19,054 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:19,054 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:19,054 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:19,054 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:19,054 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:19,054 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:19,054 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:19,054 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:19,054 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. started... 2023-03-31 16:54:19,054 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. as already flushing 2023-03-31 16:54:19,054 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:19,055 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:19,055 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:19,055 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:19,055 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:19,055 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:19,056 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:19,056 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:19,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:19,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:19,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:19,057 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:19,057 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:19,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:19,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:19,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:19,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:19,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:19,060 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:19,060 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ef9733c[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:19,060 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:19,060 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:19,061 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/0fc1625fa8f74442bb50483deca7510c, entries=150, sequenceid=118, filesize=11.6 K 2023-03-31 16:54:19,062 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/C/4e399327d7954581853ef1d4ce813b37 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/4e399327d7954581853ef1d4ce813b37 2023-03-31 16:54:19,063 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:19,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:19,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:19,063 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:19,063 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:19,063 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:19,072 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:19,072 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:19,072 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:19,072 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:19,072 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:19,072 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:19,072 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:19,072 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:19,072 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:19,072 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:19,073 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:19,073 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/4e399327d7954581853ef1d4ce813b37, entries=150, sequenceid=118, filesize=11.6 K 2023-03-31 16:54:19,074 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=147.60 KB/151140 for e76385ddbd37af190ba2df92b5602e3b in 520ms, sequenceid=118, compaction requested=true 2023-03-31 16:54:19,074 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e76385ddbd37af190ba2df92b5602e3b: 2023-03-31 16:54:19,074 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:54:19,074 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-31 16:54:19,074 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:19,074 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:54:19,074 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-31 16:54:19,074 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:54:19,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:19,074 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:19,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:19,075 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:19,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:19,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:19,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:19,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:19,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:19,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:19,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:19,076 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:19,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:19,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:19,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:19,077 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 203064 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-31 16:54:19,077 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1912): e76385ddbd37af190ba2df92b5602e3b/A is initiating minor compaction (all files) 2023-03-31 16:54:19,077 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e76385ddbd37af190ba2df92b5602e3b/A in TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:19,077 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71316 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-31 16:54:19,078 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/22abf05162964cb698f718f8d8a09162, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/caf1fcac7bcf41e99bede9c8d7470dfb, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/7125aeff4e0e423282a4bf05dc0fd495, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/30aa652862a947dfbc88722a0caa6743, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/9ae7627fbf8b4536a2a289cfae4492ed, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/a1beb4ab665c4d69a714f89128eaa846] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp, totalSize=198.3 K 2023-03-31 16:54:19,078 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): e76385ddbd37af190ba2df92b5602e3b/B is initiating minor compaction (all files) 2023-03-31 16:54:19,078 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e76385ddbd37af190ba2df92b5602e3b/B in TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:19,078 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/bb3f5d3f50584c91a07b0498a7d638eb, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/9618f2346f0e4219afe0af8176a26ca9, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/8ec4b187750e43ada713cc6e720ee32c, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/cffe11007d554e74ba6f7b376f31e142, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/b87470db93dc46e7b06273abafeb16b9, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/0fc1625fa8f74442bb50483deca7510c] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp, totalSize=69.6 K 2023-03-31 16:54:19,079 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 22abf05162964cb698f718f8d8a09162, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1680281653979 2023-03-31 16:54:19,079 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting caf1fcac7bcf41e99bede9c8d7470dfb, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1680281654051 2023-03-31 16:54:19,079 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting bb3f5d3f50584c91a07b0498a7d638eb, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1680281653979 2023-03-31 16:54:19,080 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 7125aeff4e0e423282a4bf05dc0fd495, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1680281654655 2023-03-31 16:54:19,080 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 9618f2346f0e4219afe0af8176a26ca9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1680281654051 2023-03-31 16:54:19,081 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 8ec4b187750e43ada713cc6e720ee32c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1680281654655 2023-03-31 16:54:19,081 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 30aa652862a947dfbc88722a0caa6743, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1680281656227 2023-03-31 16:54:19,082 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting cffe11007d554e74ba6f7b376f31e142, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1680281656227 2023-03-31 16:54:19,082 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 9ae7627fbf8b4536a2a289cfae4492ed, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=105, earliestPutTs=1680281657354 2023-03-31 16:54:19,082 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting b87470db93dc46e7b06273abafeb16b9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=105, earliestPutTs=1680281657354 2023-03-31 16:54:19,082 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 0fc1625fa8f74442bb50483deca7510c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=118, earliestPutTs=1680281658508 2023-03-31 16:54:19,083 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting a1beb4ab665c4d69a714f89128eaa846, keycount=400, bloomtype=ROW, size=59.0 K, encoding=NONE, compression=NONE, seqNum=118, earliestPutTs=1680281658506 2023-03-31 16:54:19,084 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:19,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:19,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:19,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:19,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:19,088 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:19,088 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:19,088 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:19,088 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:19,088 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:19,088 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:19,089 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:19,089 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:19,089 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:19,089 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:19,089 DEBUG [Thread-907] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:19,089 DEBUG [Thread-907] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:19,089 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:19,091 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:19,091 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:19,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:19,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:19,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:19,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:19,104 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): e76385ddbd37af190ba2df92b5602e3b#B#compaction#137 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:54:19,138 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] throttle.PressureAwareThroughputController(145): e76385ddbd37af190ba2df92b5602e3b#A#compaction#138 average throughput is 0.60 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:54:19,189 DEBUG [Thread-907] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:19,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:19,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:19,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:19,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:19,192 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:19,192 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:19,192 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:19,192 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:19,193 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:19,193 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:19,194 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:19,194 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:19,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:19,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:19,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:19,194 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:19,194 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:19,195 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:19,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:19,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:19,195 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:19,195 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:19,195 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:19,195 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:19,195 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:19,195 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:19,196 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:19,196 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:19,196 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:19,196 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. started... 2023-03-31 16:54:19,196 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e76385ddbd37af190ba2df92b5602e3b 3/3 column families, dataSize=147.60 KB heapSize=359.20 KB 2023-03-31 16:54:19,196 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e76385ddbd37af190ba2df92b5602e3b, store=A 2023-03-31 16:54:19,196 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:54:19,196 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e76385ddbd37af190ba2df92b5602e3b, store=B 2023-03-31 16:54:19,196 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:54:19,196 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e76385ddbd37af190ba2df92b5602e3b, store=C 2023-03-31 16:54:19,197 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:54:19,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on e76385ddbd37af190ba2df92b5602e3b 2023-03-31 16:54:19,244 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. as already flushing 2023-03-31 16:54:19,255 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:19,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52920 deadline: 1680281719252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:19,255 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:19,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52872 deadline: 1680281719253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:19,255 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:19,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52892 deadline: 1680281719254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:19,262 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:19,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52878 deadline: 1680281719255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:19,359 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:19,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52920 deadline: 1680281719356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:19,359 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:19,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52872 deadline: 1680281719356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:19,359 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:19,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52892 deadline: 1680281719356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:19,365 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:19,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52878 deadline: 1680281719363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:19,528 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/B/074f475d70214880a9bbec74f8fc49b6 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/074f475d70214880a9bbec74f8fc49b6 2023-03-31 16:54:19,533 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in e76385ddbd37af190ba2df92b5602e3b/B of e76385ddbd37af190ba2df92b5602e3b into 074f475d70214880a9bbec74f8fc49b6(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:54:19,533 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e76385ddbd37af190ba2df92b5602e3b: 2023-03-31 16:54:19,533 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b., storeName=e76385ddbd37af190ba2df92b5602e3b/B, priority=10, startTime=1680281659074; duration=0sec 2023-03-31 16:54:19,533 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:54:19,534 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-31 16:54:19,536 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71316 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-31 16:54:19,537 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): e76385ddbd37af190ba2df92b5602e3b/C is initiating minor compaction (all files) 2023-03-31 16:54:19,537 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e76385ddbd37af190ba2df92b5602e3b/C in TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:19,537 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/308dfc7bca3948e690cefe3994e99656, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/1582f5e7f9d246fa8594e5860f39e221, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/f7265421edeb4afab79c0e61a58898ca, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/0172887afb7a44b2bf66317ee3b84e47, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/329c493d8e7f4c79aa85852a3e5dcdd1, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/4e399327d7954581853ef1d4ce813b37] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp, totalSize=69.6 K 2023-03-31 16:54:19,537 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 308dfc7bca3948e690cefe3994e99656, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1680281653979 2023-03-31 16:54:19,538 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 1582f5e7f9d246fa8594e5860f39e221, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1680281654051 2023-03-31 16:54:19,539 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting f7265421edeb4afab79c0e61a58898ca, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1680281654655 2023-03-31 16:54:19,539 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 0172887afb7a44b2bf66317ee3b84e47, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1680281656227 2023-03-31 16:54:19,540 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 329c493d8e7f4c79aa85852a3e5dcdd1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=105, earliestPutTs=1680281657354 2023-03-31 16:54:19,540 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 4e399327d7954581853ef1d4ce813b37, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=118, earliestPutTs=1680281658508 2023-03-31 16:54:19,555 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): e76385ddbd37af190ba2df92b5602e3b#C#compaction#140 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:54:19,562 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:19,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52920 deadline: 1680281719560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:19,562 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:19,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52872 deadline: 1680281719561, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:19,562 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:19,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52892 deadline: 1680281719562, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:19,573 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:19,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52878 deadline: 1680281719567, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:19,577 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/C/abd6b9ba3e694e18aeebe2c33a66fe92 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/abd6b9ba3e694e18aeebe2c33a66fe92 2023-03-31 16:54:19,585 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/A/9123603f0f8b4a14a2be8620eeda5855 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/9123603f0f8b4a14a2be8620eeda5855 2023-03-31 16:54:19,586 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in e76385ddbd37af190ba2df92b5602e3b/C of e76385ddbd37af190ba2df92b5602e3b into abd6b9ba3e694e18aeebe2c33a66fe92(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:54:19,586 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e76385ddbd37af190ba2df92b5602e3b: 2023-03-31 16:54:19,586 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b., storeName=e76385ddbd37af190ba2df92b5602e3b/C, priority=10, startTime=1680281659074; duration=0sec 2023-03-31 16:54:19,586 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:54:19,591 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in e76385ddbd37af190ba2df92b5602e3b/A of e76385ddbd37af190ba2df92b5602e3b into 9123603f0f8b4a14a2be8620eeda5855(size=25.3 K), total size for store is 25.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:54:19,591 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e76385ddbd37af190ba2df92b5602e3b: 2023-03-31 16:54:19,591 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b., storeName=e76385ddbd37af190ba2df92b5602e3b/A, priority=10, startTime=1680281659074; duration=0sec 2023-03-31 16:54:19,591 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:54:19,612 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023033119aabb9ccfde4c8baf432f6076014a74 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033119aabb9ccfde4c8baf432f6076014a74 2023-03-31 16:54:19,865 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:19,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52920 deadline: 1680281719863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:19,866 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:19,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52872 deadline: 1680281719864, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:19,866 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:19,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52892 deadline: 1680281719865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:19,876 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:19,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52878 deadline: 1680281719875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:20,023 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=143, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/A/d76d45ffd225438482ce156f4161d195 2023-03-31 16:54:20,048 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=143 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/B/b1c0303b69d445a3ad1ebd4679ebbd84 2023-03-31 16:54:20,367 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:20,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52872 deadline: 1680281720367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:20,368 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:20,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52920 deadline: 1680281720368, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:20,369 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:20,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52892 deadline: 1680281720368, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:20,380 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:20,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52878 deadline: 1680281720377, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:20,479 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=143 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/C/644f985a1b5c41f1a493577b6829d5b0 2023-03-31 16:54:20,484 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/A/d76d45ffd225438482ce156f4161d195 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/d76d45ffd225438482ce156f4161d195 2023-03-31 16:54:20,489 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/d76d45ffd225438482ce156f4161d195, entries=150, sequenceid=143, filesize=25.3 K 2023-03-31 16:54:20,490 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/B/b1c0303b69d445a3ad1ebd4679ebbd84 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/b1c0303b69d445a3ad1ebd4679ebbd84 2023-03-31 16:54:20,494 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/b1c0303b69d445a3ad1ebd4679ebbd84, entries=150, sequenceid=143, filesize=11.8 K 2023-03-31 16:54:20,495 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/C/644f985a1b5c41f1a493577b6829d5b0 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/644f985a1b5c41f1a493577b6829d5b0 2023-03-31 16:54:20,499 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/644f985a1b5c41f1a493577b6829d5b0, entries=150, sequenceid=143, filesize=11.8 K 2023-03-31 16:54:20,499 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~359.16 KB/367776, currentSize=53.67 KB/54960 for e76385ddbd37af190ba2df92b5602e3b in 1303ms, sequenceid=143, compaction requested=false 2023-03-31 16:54:20,500 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e76385ddbd37af190ba2df92b5602e3b: 2023-03-31 16:54:20,500 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:20,500 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:20,500 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:20,500 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:20,500 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:20,500 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:20,502 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:20,502 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:20,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:20,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:20,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:20,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:20,502 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:20,502 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:20,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:20,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:20,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:20,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:20,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:20,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47ee94c4[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:20,504 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:20,504 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:20,505 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:20,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:20,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:20,505 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:20,505 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:20,505 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:20,506 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:20,506 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:20,507 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:20,507 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:20,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:20,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:20,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:20,507 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:20,508 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:20,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:20,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:20,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:20,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:20,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:20,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:20,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:20,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:20,510 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:20,510 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:20,510 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:20,510 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:20,510 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:20,510 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:20,512 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:20,512 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:20,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:20,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:20,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:20,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:20,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:20,512 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:20,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:20,512 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:20,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:20,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:20,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:20,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:20,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:20,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:20,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:20,514 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:20,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:20,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:20,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:20,516 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:20,516 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:20,516 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:20,516 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:20,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:20,516 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:20,516 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:20,516 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:20,516 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:20,516 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:20,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:20,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:20,516 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:20,517 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:20,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:20,517 DEBUG [Thread-907] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:20,517 DEBUG [Thread-907] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:20,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:20,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:20,617 DEBUG [Thread-907] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:20,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:20,619 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:20,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:20,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:20,621 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:20,621 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:20,621 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:20,621 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:20,622 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:20,622 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:20,623 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:20,623 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:20,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:20,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:20,624 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:20,624 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:20,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:20,624 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:20,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:20,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:20,625 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:20,625 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:20,625 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:20,625 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:20,625 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:20,625 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:20,625 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:20,625 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:20,625 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:20,625 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. started... 2023-03-31 16:54:20,626 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e76385ddbd37af190ba2df92b5602e3b 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-31 16:54:20,626 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e76385ddbd37af190ba2df92b5602e3b, store=A 2023-03-31 16:54:20,626 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:20,626 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e76385ddbd37af190ba2df92b5602e3b, store=B 2023-03-31 16:54:20,626 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:20,626 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e76385ddbd37af190ba2df92b5602e3b, store=C 2023-03-31 16:54:20,626 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:21,054 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303317aa5b56792e445979f8edf9984c6fdb7 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303317aa5b56792e445979f8edf9984c6fdb7 2023-03-31 16:54:21,062 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=157, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/A/0a78cf1961d043a59eba0b61848a3f43 2023-03-31 16:54:21,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on e76385ddbd37af190ba2df92b5602e3b 2023-03-31 16:54:21,373 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. as already flushing 2023-03-31 16:54:21,397 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680281661387, max timestamp=1680281661395Num uniques 859; 2023-03-31 16:54:21,397 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-31 16:54:21,397 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680281661387, max timestamp=1680281661395Num uniques 859; 2023-03-31 16:54:21,397 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-31 16:54:21,397 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680281661387, max timestamp=1680281661395Num uniques 859; 2023-03-31 16:54:21,397 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-31 16:54:21,412 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:21,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52872 deadline: 1680281721407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:21,412 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:21,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52878 deadline: 1680281721410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:21,413 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:21,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52920 deadline: 1680281721411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:21,413 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:21,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52892 deadline: 1680281721412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:21,492 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=157 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/B/8239929e2581489190769f99bb68cbb3 2023-03-31 16:54:21,503 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=157 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/C/ce3235521bee48aba86f8b30ae6cd971 2023-03-31 16:54:21,509 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/A/0a78cf1961d043a59eba0b61848a3f43 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/0a78cf1961d043a59eba0b61848a3f43 2023-03-31 16:54:21,514 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/0a78cf1961d043a59eba0b61848a3f43, entries=150, sequenceid=157, filesize=25.3 K 2023-03-31 16:54:21,516 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/B/8239929e2581489190769f99bb68cbb3 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/8239929e2581489190769f99bb68cbb3 2023-03-31 16:54:21,518 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:21,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52872 deadline: 1680281721513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:21,518 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:21,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52878 deadline: 1680281721514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:21,518 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:21,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52920 deadline: 1680281721514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:21,518 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:21,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52892 deadline: 1680281721515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:21,523 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/8239929e2581489190769f99bb68cbb3, entries=150, sequenceid=157, filesize=11.8 K 2023-03-31 16:54:21,524 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/C/ce3235521bee48aba86f8b30ae6cd971 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/ce3235521bee48aba86f8b30ae6cd971 2023-03-31 16:54:21,528 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/ce3235521bee48aba86f8b30ae6cd971, entries=150, sequenceid=157, filesize=11.8 K 2023-03-31 16:54:21,529 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for e76385ddbd37af190ba2df92b5602e3b in 903ms, sequenceid=157, compaction requested=true 2023-03-31 16:54:21,529 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e76385ddbd37af190ba2df92b5602e3b: 2023-03-31 16:54:21,529 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:21,529 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:21,529 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:21,529 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:21,529 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:21,529 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:21,531 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:21,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:21,531 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:21,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:21,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:21,532 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:21,532 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:21,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:21,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:21,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:21,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:21,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:21,535 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:21,535 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3eda3a5c[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:21,535 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:21,535 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:21,536 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:21,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:21,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:21,536 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:21,536 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:21,536 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:21,536 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:21,536 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:21,537 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:21,537 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:21,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:21,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:21,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:21,537 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:21,538 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:21,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:21,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:21,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:21,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:21,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:21,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:21,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:21,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:21,540 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:21,540 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:21,540 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:21,540 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:21,540 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:21,540 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:21,543 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:21,543 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:21,543 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:21,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:21,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:21,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:21,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:21,544 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:21,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:21,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:21,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:21,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:21,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:21,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:21,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:21,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:21,545 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:21,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:21,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:21,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:21,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:21,547 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:21,547 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:21,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:21,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:21,547 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:21,547 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:21,547 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:21,548 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:21,548 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:21,548 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:21,548 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:21,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:21,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:21,547 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:21,548 DEBUG [Thread-907] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:21,547 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:21,548 DEBUG [Thread-907] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:21,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:21,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:21,648 DEBUG [Thread-907] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:21,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:21,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:21,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:21,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:21,653 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:21,653 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:21,653 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:21,653 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:21,654 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:21,654 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:21,656 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:21,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:21,656 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:21,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:21,657 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:21,657 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:21,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:21,657 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:21,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:21,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:21,658 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:21,658 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:21,658 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:21,658 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:21,658 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:21,658 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:21,658 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:21,658 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:21,658 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:21,658 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. started... 2023-03-31 16:54:21,659 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e76385ddbd37af190ba2df92b5602e3b 3/3 column families, dataSize=134.18 KB heapSize=370.92 KB 2023-03-31 16:54:21,659 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e76385ddbd37af190ba2df92b5602e3b, store=A 2023-03-31 16:54:21,659 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:54:21,659 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e76385ddbd37af190ba2df92b5602e3b, store=B 2023-03-31 16:54:21,659 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:54:21,659 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e76385ddbd37af190ba2df92b5602e3b, store=C 2023-03-31 16:54:21,659 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:54:21,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on e76385ddbd37af190ba2df92b5602e3b 2023-03-31 16:54:21,722 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. as already flushing 2023-03-31 16:54:21,732 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:21,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52872 deadline: 1680281721727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:21,733 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:21,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52920 deadline: 1680281721728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:21,733 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:21,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52878 deadline: 1680281721729, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:21,733 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:21,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52892 deadline: 1680281721730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:21,834 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:21,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52872 deadline: 1680281721833, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:21,835 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:21,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52920 deadline: 1680281721834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:21,835 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:21,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52878 deadline: 1680281721834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:21,837 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:21,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52892 deadline: 1680281721834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:22,036 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:22,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52872 deadline: 1680281722036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:22,039 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:22,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52920 deadline: 1680281722037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:22,040 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:22,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52878 deadline: 1680281722037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:22,042 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:22,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52892 deadline: 1680281722039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:22,087 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230331cd717c6f0dbe40a0afd249730471ab01 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331cd717c6f0dbe40a0afd249730471ab01 2023-03-31 16:54:22,346 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:22,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52878 deadline: 1680281722342, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:22,346 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:22,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52920 deadline: 1680281722343, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:22,347 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:22,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52872 deadline: 1680281722343, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:22,347 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:22,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52892 deadline: 1680281722344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:22,492 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=191, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/A/c0e7d3daf0b04c6cab9c84a9c888b3cd 2023-03-31 16:54:22,504 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=191 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/B/7aec5dbcc7034377965cdb4265fe079d 2023-03-31 16:54:22,812 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:22,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52906 deadline: 1680281722810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:22,813 DEBUG [Thread-905] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8150 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b., hostname=jenkins-hbase4.apache.org,33441,1680281619375, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-31 16:54:22,851 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:22,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52920 deadline: 1680281722849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:22,851 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:22,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52878 deadline: 1680281722849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:22,851 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:22,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52892 deadline: 1680281722849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:22,851 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:22,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52872 deadline: 1680281722851, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:22,919 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=191 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/C/be809c5544aa4162b2352f4a0b24f679 2023-03-31 16:54:22,924 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/A/c0e7d3daf0b04c6cab9c84a9c888b3cd as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/c0e7d3daf0b04c6cab9c84a9c888b3cd 2023-03-31 16:54:22,928 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/c0e7d3daf0b04c6cab9c84a9c888b3cd, entries=150, sequenceid=191, filesize=25.3 K 2023-03-31 16:54:22,929 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/B/7aec5dbcc7034377965cdb4265fe079d as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/7aec5dbcc7034377965cdb4265fe079d 2023-03-31 16:54:22,933 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/7aec5dbcc7034377965cdb4265fe079d, entries=150, sequenceid=191, filesize=11.8 K 2023-03-31 16:54:22,934 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/C/be809c5544aa4162b2352f4a0b24f679 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/be809c5544aa4162b2352f4a0b24f679 2023-03-31 16:54:22,938 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/be809c5544aa4162b2352f4a0b24f679, entries=150, sequenceid=191, filesize=11.8 K 2023-03-31 16:54:22,939 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~370.88 KB/379776, currentSize=53.67 KB/54960 for e76385ddbd37af190ba2df92b5602e3b in 1280ms, sequenceid=191, compaction requested=true 2023-03-31 16:54:22,939 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e76385ddbd37af190ba2df92b5602e3b: 2023-03-31 16:54:22,939 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:22,939 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:22,939 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:22,939 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:22,939 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:22,939 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:22,941 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:22,941 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:22,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:22,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:22,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:22,941 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:22,941 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:22,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:22,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:22,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:22,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:22,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:22,942 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:22,942 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5500be7c[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:22,942 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:22,942 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:22,944 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:22,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:22,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:22,945 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:22,945 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:22,945 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:22,945 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:22,945 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:22,946 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:22,946 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:22,946 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:22,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:22,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:22,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:22,946 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:22,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:22,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:22,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:22,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:22,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:22,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:22,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:22,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:22,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:22,949 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:22,949 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:22,949 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:22,949 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:22,949 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:22,953 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:22,953 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:22,953 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:22,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:22,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:22,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:22,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:22,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:22,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:22,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:22,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:22,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:22,954 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:22,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:22,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:22,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:22,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:22,956 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:22,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:22,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:22,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:22,959 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:22,959 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:22,959 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:22,959 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:22,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:22,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:22,959 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:22,959 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:22,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:22,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:22,959 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:22,960 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:22,960 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:22,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:22,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:22,960 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:22,960 DEBUG [Thread-907] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:22,960 DEBUG [Thread-907] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:23,060 DEBUG [Thread-907] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:23,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:23,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:23,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:23,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:23,063 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:23,064 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:23,064 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:23,064 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:23,064 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:23,064 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:23,065 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:23,065 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:23,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:23,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:23,066 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:23,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:23,066 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:23,066 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:23,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:23,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:23,066 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:23,067 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:23,067 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:23,067 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:23,067 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:23,067 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:23,067 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:23,067 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:23,067 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:23,067 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. started... 2023-03-31 16:54:23,067 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e76385ddbd37af190ba2df92b5602e3b 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-31 16:54:23,067 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e76385ddbd37af190ba2df92b5602e3b, store=A 2023-03-31 16:54:23,068 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:23,068 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e76385ddbd37af190ba2df92b5602e3b, store=B 2023-03-31 16:54:23,068 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:23,068 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e76385ddbd37af190ba2df92b5602e3b, store=C 2023-03-31 16:54:23,068 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:23,080 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023033152ea7abf49944c019d18b642fe18b295 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033152ea7abf49944c019d18b642fe18b295 2023-03-31 16:54:23,096 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=202, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/A/c8527ce4bbff417daff8269fbdd037d8 2023-03-31 16:54:23,522 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=202 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/B/b21f40184b63425283a74bb12189bf64 2023-03-31 16:54:23,542 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=202 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/C/1b0e3918c6f04390b5dc193c87fee2aa 2023-03-31 16:54:23,548 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/A/c8527ce4bbff417daff8269fbdd037d8 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/c8527ce4bbff417daff8269fbdd037d8 2023-03-31 16:54:23,552 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/c8527ce4bbff417daff8269fbdd037d8, entries=150, sequenceid=202, filesize=25.3 K 2023-03-31 16:54:23,553 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/B/b21f40184b63425283a74bb12189bf64 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/b21f40184b63425283a74bb12189bf64 2023-03-31 16:54:23,557 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/b21f40184b63425283a74bb12189bf64, entries=150, sequenceid=202, filesize=11.8 K 2023-03-31 16:54:23,558 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/C/1b0e3918c6f04390b5dc193c87fee2aa as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/1b0e3918c6f04390b5dc193c87fee2aa 2023-03-31 16:54:23,567 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/1b0e3918c6f04390b5dc193c87fee2aa, entries=150, sequenceid=202, filesize=11.8 K 2023-03-31 16:54:23,568 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=0 B/0 for e76385ddbd37af190ba2df92b5602e3b in 501ms, sequenceid=202, compaction requested=true 2023-03-31 16:54:23,568 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e76385ddbd37af190ba2df92b5602e3b: 2023-03-31 16:54:23,568 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:23,568 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:23,568 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:23,568 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:23,568 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:23,568 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:23,571 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:23,571 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:23,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:23,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:23,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:23,584 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:23,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:23,584 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:23,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:23,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:23,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:23,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:23,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:23,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b54773[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:23,586 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:23,586 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:23,587 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:23,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:23,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:23,588 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:23,588 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:23,588 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:23,588 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:23,588 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:23,589 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:23,589 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:23,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:23,590 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:23,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:23,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:23,590 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:23,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:23,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:23,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:23,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:23,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:23,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:23,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:23,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:23,592 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:23,593 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:23,593 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:23,593 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:23,593 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:23,593 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:23,595 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:23,595 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:23,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:23,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:23,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:23,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:23,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:23,595 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:23,596 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:23,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:23,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:23,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:23,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:23,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:23,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:23,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:23,597 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:23,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:23,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:23,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:23,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:23,601 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:23,601 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:23,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:23,602 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:23,602 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:23,601 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:23,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:23,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:23,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:23,602 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:23,602 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:23,602 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:23,602 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:23,602 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:23,602 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:23,602 DEBUG [Thread-907] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:23,602 DEBUG [Thread-907] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:23,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:23,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:23,702 DEBUG [Thread-907] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:23,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:23,705 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:23,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:23,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:23,707 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:23,707 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:23,707 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:23,707 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:23,707 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:23,707 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:23,709 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:23,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:23,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:23,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:23,709 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:23,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:23,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:23,710 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:23,710 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:23,710 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:23,710 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:23,710 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:23,711 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:23,711 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:23,711 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:23,712 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:23,712 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:23,712 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:23,713 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. started... 2023-03-31 16:54:23,713 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e76385ddbd37af190ba2df92b5602e3b: 2023-03-31 16:54:23,713 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:23,713 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:23,713 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:23,713 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:23,713 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:23,713 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:23,714 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:23,714 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:23,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:23,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:23,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:23,715 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:23,715 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:23,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:23,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:23,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:23,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:23,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:23,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:23,717 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:23,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@334eab15[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:23,717 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:23,718 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:23,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:23,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:23,718 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:23,718 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:23,718 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:23,718 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:23,718 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:23,721 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:23,721 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:23,721 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:23,721 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:23,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:23,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:23,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:23,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:23,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:23,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:23,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:23,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:23,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:23,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:23,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:23,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:23,724 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:23,724 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:23,724 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:23,724 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:23,724 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:23,726 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:23,726 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:23,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:23,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:23,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:23,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:23,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:23,726 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:23,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:23,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:23,727 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:23,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:23,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:23,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:23,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:23,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:23,729 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:23,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:23,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:23,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:23,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:23,731 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:23,731 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:23,731 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:23,731 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:23,731 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:23,731 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:23,731 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:23,731 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:23,731 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:23,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:23,732 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:23,731 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:23,731 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:23,732 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:23,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:23,732 DEBUG [Thread-907] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:23,732 DEBUG [Thread-907] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:23,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:23,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:23,832 DEBUG [Thread-907] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:23,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:23,834 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:23,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:23,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:23,855 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:23,855 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:23,855 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:23,855 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:23,856 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:23,856 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:23,857 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:23,857 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:23,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:23,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:23,857 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:23,857 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:23,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:23,858 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:23,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:23,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:23,859 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:23,872 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:23,872 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:23,872 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:23,872 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:23,872 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:23,873 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:23,873 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:23,873 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:23,873 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. started... 2023-03-31 16:54:23,873 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e76385ddbd37af190ba2df92b5602e3b: 2023-03-31 16:54:23,873 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:23,873 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:23,873 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:23,873 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:23,873 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:23,873 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:23,876 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:23,876 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:23,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:23,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:23,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:23,877 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:23,877 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:23,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:23,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:23,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:23,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:23,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:23,879 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:23,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c297ffa[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:23,880 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:23,880 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:23,881 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:23,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:23,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:23,881 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:23,881 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:23,882 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:23,882 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:23,882 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:23,883 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:23,883 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:23,883 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:23,883 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:23,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:23,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:23,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:23,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:23,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:23,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:23,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:23,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:23,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:23,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:23,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:23,887 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:23,887 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:23,887 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:23,887 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:23,887 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:23,887 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:23,888 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:23,888 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:23,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:23,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:23,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:23,888 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:23,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:23,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:23,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:23,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:23,889 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:23,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:23,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:23,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:23,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:23,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:23,891 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:23,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:23,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:23,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:23,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:23,897 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:23,897 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:23,897 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:23,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:23,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:23,897 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:23,898 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:23,898 DEBUG [Thread-907] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:23,898 DEBUG [Thread-907] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:23,897 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:23,897 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:23,898 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:23,898 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:23,898 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:23,898 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:23,898 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:23,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:23,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:23,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on e76385ddbd37af190ba2df92b5602e3b 2023-03-31 16:54:23,904 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e76385ddbd37af190ba2df92b5602e3b 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-31 16:54:23,905 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e76385ddbd37af190ba2df92b5602e3b, store=A 2023-03-31 16:54:23,905 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:23,905 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e76385ddbd37af190ba2df92b5602e3b, store=B 2023-03-31 16:54:23,906 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:23,906 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e76385ddbd37af190ba2df92b5602e3b, store=C 2023-03-31 16:54:23,906 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:23,936 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303318adf260487eb47f6b4b3f5b6317af52c to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303318adf260487eb47f6b4b3f5b6317af52c 2023-03-31 16:54:23,956 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=215, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/A/62c7371e752d4e0388488a49fc7c2844 2023-03-31 16:54:23,958 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1680281663945, max timestamp=1680281663955Num uniques 859; 2023-03-31 16:54:23,958 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 16:54:23,959 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1680281663945, max timestamp=1680281663955Num uniques 859; 2023-03-31 16:54:23,959 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 16:54:23,960 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680281663945, max timestamp=1680281663955Num uniques 859; 2023-03-31 16:54:23,960 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 16:54:23,982 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=215 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/B/08fdc2d8e0ac4355bd611f9c1370ece5 2023-03-31 16:54:23,983 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:23,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52892 deadline: 1680281723975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:23,986 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:23,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52872 deadline: 1680281723976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:23,986 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:23,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52920 deadline: 1680281723976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:23,986 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:23,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52878 deadline: 1680281723977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:23,998 DEBUG [Thread-907] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:23,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:24,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:24,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:24,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:24,004 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:24,004 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:24,004 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:24,004 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:24,005 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:24,005 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:24,006 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,006 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:24,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:24,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:24,006 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,006 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:24,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:24,007 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:24,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:24,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:24,007 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:24,007 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:24,007 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:24,007 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:24,008 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:24,008 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:24,008 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:24,008 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:24,008 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:24,008 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. started... 2023-03-31 16:54:24,008 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. as already flushing 2023-03-31 16:54:24,008 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:24,008 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:24,008 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:24,008 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,008 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:24,008 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:24,010 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:24,010 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:24,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:24,010 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:24,010 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:24,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:24,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:24,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:24,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:24,012 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:24,012 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c8ad95d[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:24,012 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:24,012 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:24,013 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:24,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:24,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:24,013 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:24,014 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:24,014 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,014 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:24,014 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:24,016 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:24,016 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,016 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:24,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:24,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:24,016 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:24,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:24,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:24,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:24,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:24,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:24,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:24,021 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:24,021 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:24,021 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:24,021 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:24,021 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:24,021 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:24,023 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:24,023 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:24,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:24,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:24,023 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:24,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:24,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:24,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:24,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:24,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:24,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:24,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:24,025 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:24,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:24,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:24,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,026 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:24,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:24,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,029 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:24,029 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:24,029 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,029 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:24,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:24,029 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:24,029 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:24,029 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,029 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:24,029 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:24,029 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:24,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:24,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:24,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:24,030 DEBUG [Thread-907] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:24,030 DEBUG [Thread-907] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:24,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:24,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:24,088 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:24,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52892 deadline: 1680281724084, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,091 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:24,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52872 deadline: 1680281724087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,092 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:24,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52920 deadline: 1680281724087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,092 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:24,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52878 deadline: 1680281724088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,130 DEBUG [Thread-907] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:24,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:24,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:24,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:24,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:24,134 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:24,134 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:24,134 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:24,134 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:24,135 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:24,135 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:24,136 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:24,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:24,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:24,136 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:24,137 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,137 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:24,137 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:24,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:24,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:24,138 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:24,138 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:24,138 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:24,138 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:24,138 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:24,138 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:24,139 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:24,139 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:24,139 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:24,139 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. started... 2023-03-31 16:54:24,140 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. as already flushing 2023-03-31 16:54:24,140 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:24,140 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:24,140 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:24,140 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,140 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:24,140 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:24,143 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,143 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:24,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:24,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:24,144 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:24,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:24,144 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:24,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:24,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:24,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:24,145 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:24,145 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@262e8960[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:24,146 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:24,146 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:24,147 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,147 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:24,148 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:24,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:24,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:24,148 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:24,148 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:24,148 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:24,150 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:24,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:24,150 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:24,150 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:24,150 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:24,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:24,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:24,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:24,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:24,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:24,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:24,154 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:24,154 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:24,154 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:24,154 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:24,154 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:24,154 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:24,157 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:24,157 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:24,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:24,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:24,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:24,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:24,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:24,157 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:24,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:24,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:24,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:24,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:24,158 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:24,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:24,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:24,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,160 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:24,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:24,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,163 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,163 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:24,163 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:24,163 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:24,163 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:24,163 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:24,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:24,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:24,163 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:24,164 DEBUG [Thread-907] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:24,164 DEBUG [Thread-907] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:24,163 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:24,163 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:24,164 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:24,164 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:24,164 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:24,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:24,264 DEBUG [Thread-907] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:24,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:24,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:24,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:24,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:24,267 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:24,267 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:24,268 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:24,268 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:24,268 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:24,268 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:24,269 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,269 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:24,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:24,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:24,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:24,272 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:24,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:24,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:24,272 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:24,272 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:24,272 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:24,273 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:24,273 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:24,273 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:24,273 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:24,273 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:24,273 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:24,273 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. started... 2023-03-31 16:54:24,273 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. as already flushing 2023-03-31 16:54:24,273 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:24,273 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:24,273 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:24,274 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,274 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:24,274 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:24,276 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:24,276 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:24,276 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:24,280 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,280 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,280 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:24,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:24,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:24,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:24,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:24,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:24,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:24,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:24,284 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:24,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@607e98b3[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:24,284 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:24,286 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:24,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:24,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:24,287 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:24,287 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:24,287 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:24,287 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,287 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:24,288 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:24,288 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:24,289 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:24,289 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:24,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:24,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:24,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:24,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:24,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:24,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:24,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:24,292 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:24,292 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:24,292 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:24,292 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:24,292 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:24,292 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:24,294 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:24,294 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:24,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:24,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:24,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:24,294 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:24,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:24,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:24,294 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:24,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52892 deadline: 1680281724289, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:24,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:24,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:24,295 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:24,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:24,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:24,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:24,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,297 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:24,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:24,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,298 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:24,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52920 deadline: 1680281724293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,299 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:24,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52872 deadline: 1680281724293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,299 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:24,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52878 deadline: 1680281724294, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,304 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,304 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:24,304 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:24,304 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,304 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:24,304 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:24,304 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:24,304 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:24,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:24,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:24,304 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:24,304 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:24,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:24,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:24,305 DEBUG [Thread-907] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:24,305 DEBUG [Thread-907] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:24,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:24,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:24,395 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=215 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/C/a82e2b27555345279a58674ad41912ed 2023-03-31 16:54:24,403 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/A/62c7371e752d4e0388488a49fc7c2844 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/62c7371e752d4e0388488a49fc7c2844 2023-03-31 16:54:24,405 DEBUG [Thread-907] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:24,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:24,408 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/62c7371e752d4e0388488a49fc7c2844, entries=300, sequenceid=215, filesize=45.8 K 2023-03-31 16:54:24,408 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:24,410 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/B/08fdc2d8e0ac4355bd611f9c1370ece5 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/08fdc2d8e0ac4355bd611f9c1370ece5 2023-03-31 16:54:24,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:24,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:24,410 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:24,410 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:24,410 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:24,410 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:24,411 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:24,411 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:24,412 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:24,412 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:24,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:24,412 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,413 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:24,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:24,413 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:24,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:24,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:24,414 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:24,414 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:24,414 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:24,414 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:24,414 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:24,414 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:24,414 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:24,414 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:24,414 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:24,414 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. started... 2023-03-31 16:54:24,414 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. as already flushing 2023-03-31 16:54:24,414 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:24,415 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:24,415 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:24,415 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,415 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:24,415 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/08fdc2d8e0ac4355bd611f9c1370ece5, entries=150, sequenceid=215, filesize=11.8 K 2023-03-31 16:54:24,415 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:24,416 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/C/a82e2b27555345279a58674ad41912ed as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/a82e2b27555345279a58674ad41912ed 2023-03-31 16:54:24,416 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,416 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:24,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:24,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:24,417 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:24,417 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:24,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:24,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:24,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:24,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:24,420 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:24,420 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77bfede9[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:24,420 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:24,420 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:24,420 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/a82e2b27555345279a58674ad41912ed, entries=150, sequenceid=215, filesize=11.8 K 2023-03-31 16:54:24,421 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for e76385ddbd37af190ba2df92b5602e3b in 517ms, sequenceid=215, compaction requested=true 2023-03-31 16:54:24,421 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e76385ddbd37af190ba2df92b5602e3b: 2023-03-31 16:54:24,422 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:54:24,422 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:54:24,422 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-31 16:54:24,422 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-31 16:54:24,422 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-31 16:54:24,424 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 176406 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-31 16:54:24,424 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): e76385ddbd37af190ba2df92b5602e3b/A is initiating minor compaction (all files) 2023-03-31 16:54:24,424 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e76385ddbd37af190ba2df92b5602e3b/A in TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:24,424 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 72272 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-31 16:54:24,424 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/9123603f0f8b4a14a2be8620eeda5855, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/d76d45ffd225438482ce156f4161d195, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/0a78cf1961d043a59eba0b61848a3f43, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/c0e7d3daf0b04c6cab9c84a9c888b3cd, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/c8527ce4bbff417daff8269fbdd037d8, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/62c7371e752d4e0388488a49fc7c2844] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp, totalSize=172.3 K 2023-03-31 16:54:24,424 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1912): e76385ddbd37af190ba2df92b5602e3b/B is initiating minor compaction (all files) 2023-03-31 16:54:24,424 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e76385ddbd37af190ba2df92b5602e3b/B in TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:24,424 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/074f475d70214880a9bbec74f8fc49b6, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/b1c0303b69d445a3ad1ebd4679ebbd84, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/8239929e2581489190769f99bb68cbb3, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/7aec5dbcc7034377965cdb4265fe079d, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/b21f40184b63425283a74bb12189bf64, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/08fdc2d8e0ac4355bd611f9c1370ece5] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp, totalSize=70.6 K 2023-03-31 16:54:24,425 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:24,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:24,425 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,425 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:24,425 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 074f475d70214880a9bbec74f8fc49b6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=118, earliestPutTs=1680281658508 2023-03-31 16:54:24,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:24,425 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:24,425 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:24,425 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:24,425 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 9123603f0f8b4a14a2be8620eeda5855, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=118, earliestPutTs=1680281658508 2023-03-31 16:54:24,426 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting b1c0303b69d445a3ad1ebd4679ebbd84, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=143, earliestPutTs=1680281658593 2023-03-31 16:54:24,426 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting d76d45ffd225438482ce156f4161d195, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=143, earliestPutTs=1680281658593 2023-03-31 16:54:24,428 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:24,428 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:24,428 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:24,428 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:24,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:24,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:24,429 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 8239929e2581489190769f99bb68cbb3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1680281659252 2023-03-31 16:54:24,429 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 0a78cf1961d043a59eba0b61848a3f43, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1680281659252 2023-03-31 16:54:24,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:24,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:24,429 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 7aec5dbcc7034377965cdb4265fe079d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=191, earliestPutTs=1680281661409 2023-03-31 16:54:24,430 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting c0e7d3daf0b04c6cab9c84a9c888b3cd, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=191, earliestPutTs=1680281661409 2023-03-31 16:54:24,430 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting b21f40184b63425283a74bb12189bf64, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=202, earliestPutTs=1680281661725 2023-03-31 16:54:24,430 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting c8527ce4bbff417daff8269fbdd037d8, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=202, earliestPutTs=1680281661725 2023-03-31 16:54:24,430 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 08fdc2d8e0ac4355bd611f9c1370ece5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1680281663882 2023-03-31 16:54:24,430 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 62c7371e752d4e0388488a49fc7c2844, keycount=300, bloomtype=ROW, size=45.8 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1680281663882 2023-03-31 16:54:24,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:24,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:24,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:24,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:24,435 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:24,435 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:24,435 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:24,435 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:24,435 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:24,438 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:24,438 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:24,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:24,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:24,438 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:24,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:24,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:24,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:24,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:24,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:24,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:24,439 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:24,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:24,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:24,441 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:24,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:24,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:24,444 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:24,444 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:24,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:24,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:24,444 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,444 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:24,444 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:24,444 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,444 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:24,444 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:24,444 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:24,444 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:24,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:24,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:24,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:24,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:24,444 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:24,444 DEBUG [Thread-907] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:24,445 DEBUG [Thread-907] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:24,445 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] throttle.PressureAwareThroughputController(145): e76385ddbd37af190ba2df92b5602e3b#B#compaction#155 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:54:24,456 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): e76385ddbd37af190ba2df92b5602e3b#A#compaction#156 average throughput is 2.46 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:54:24,458 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/B/e0e35469f2904e61bc0de0b657e04362 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/e0e35469f2904e61bc0de0b657e04362 2023-03-31 16:54:24,463 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in e76385ddbd37af190ba2df92b5602e3b/B of e76385ddbd37af190ba2df92b5602e3b into e0e35469f2904e61bc0de0b657e04362(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:54:24,463 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e76385ddbd37af190ba2df92b5602e3b: 2023-03-31 16:54:24,463 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b., storeName=e76385ddbd37af190ba2df92b5602e3b/B, priority=10, startTime=1680281664422; duration=0sec 2023-03-31 16:54:24,463 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:54:24,463 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-31 16:54:24,482 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 72272 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-31 16:54:24,484 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1912): e76385ddbd37af190ba2df92b5602e3b/C is initiating minor compaction (all files) 2023-03-31 16:54:24,484 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e76385ddbd37af190ba2df92b5602e3b/C in TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:24,484 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/abd6b9ba3e694e18aeebe2c33a66fe92, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/644f985a1b5c41f1a493577b6829d5b0, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/ce3235521bee48aba86f8b30ae6cd971, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/be809c5544aa4162b2352f4a0b24f679, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/1b0e3918c6f04390b5dc193c87fee2aa, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/a82e2b27555345279a58674ad41912ed] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp, totalSize=70.6 K 2023-03-31 16:54:24,485 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting abd6b9ba3e694e18aeebe2c33a66fe92, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=118, earliestPutTs=1680281658508 2023-03-31 16:54:24,486 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 644f985a1b5c41f1a493577b6829d5b0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=143, earliestPutTs=1680281658593 2023-03-31 16:54:24,486 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting ce3235521bee48aba86f8b30ae6cd971, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1680281659252 2023-03-31 16:54:24,487 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting be809c5544aa4162b2352f4a0b24f679, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=191, earliestPutTs=1680281661409 2023-03-31 16:54:24,488 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 1b0e3918c6f04390b5dc193c87fee2aa, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=202, earliestPutTs=1680281661725 2023-03-31 16:54:24,488 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting a82e2b27555345279a58674ad41912ed, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1680281663882 2023-03-31 16:54:24,505 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] throttle.PressureAwareThroughputController(145): e76385ddbd37af190ba2df92b5602e3b#C#compaction#157 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:54:24,528 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/C/33c0856c380343b488cda9c1d784fe0d as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/33c0856c380343b488cda9c1d784fe0d 2023-03-31 16:54:24,535 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in e76385ddbd37af190ba2df92b5602e3b/C of e76385ddbd37af190ba2df92b5602e3b into 33c0856c380343b488cda9c1d784fe0d(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:54:24,535 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e76385ddbd37af190ba2df92b5602e3b: 2023-03-31 16:54:24,535 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b., storeName=e76385ddbd37af190ba2df92b5602e3b/C, priority=10, startTime=1680281664422; duration=0sec 2023-03-31 16:54:24,536 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:54:24,545 DEBUG [Thread-907] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:24,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:24,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:24,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:24,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:24,548 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:24,548 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:24,549 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:24,549 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:24,549 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:24,549 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:24,551 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:24,551 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:24,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:24,552 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,552 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:24,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:24,552 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:24,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:24,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:24,552 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:24,553 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:24,553 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:24,553 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:24,553 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:24,553 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:24,556 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:24,556 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:24,556 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:24,556 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. started... 2023-03-31 16:54:24,556 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e76385ddbd37af190ba2df92b5602e3b 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-03-31 16:54:24,556 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e76385ddbd37af190ba2df92b5602e3b, store=A 2023-03-31 16:54:24,556 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:54:24,557 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e76385ddbd37af190ba2df92b5602e3b, store=B 2023-03-31 16:54:24,557 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:54:24,557 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e76385ddbd37af190ba2df92b5602e3b, store=C 2023-03-31 16:54:24,557 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:54:24,587 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230331fc57b62b5069462491749dbb371249d5 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331fc57b62b5069462491749dbb371249d5 2023-03-31 16:54:24,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on e76385ddbd37af190ba2df92b5602e3b 2023-03-31 16:54:24,598 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. as already flushing 2023-03-31 16:54:24,607 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=252, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/A/08825b5aafa943f19831152a4a83b070 2023-03-31 16:54:24,611 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:24,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52872 deadline: 1680281724606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,613 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:24,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52892 deadline: 1680281724608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,615 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:24,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52878 deadline: 1680281724610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,615 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:24,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52920 deadline: 1680281724612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,626 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=252 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/B/1f136f02b8db48448907ae84cb9a687c 2023-03-31 16:54:24,650 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=252 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/C/34b28e01df4347a3a3f785a681cbf748 2023-03-31 16:54:24,656 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/A/08825b5aafa943f19831152a4a83b070 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/08825b5aafa943f19831152a4a83b070 2023-03-31 16:54:24,662 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/08825b5aafa943f19831152a4a83b070, entries=150, sequenceid=252, filesize=25.3 K 2023-03-31 16:54:24,663 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/B/1f136f02b8db48448907ae84cb9a687c as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/1f136f02b8db48448907ae84cb9a687c 2023-03-31 16:54:24,668 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/1f136f02b8db48448907ae84cb9a687c, entries=150, sequenceid=252, filesize=11.8 K 2023-03-31 16:54:24,668 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/C/34b28e01df4347a3a3f785a681cbf748 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/34b28e01df4347a3a3f785a681cbf748 2023-03-31 16:54:24,672 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/34b28e01df4347a3a3f785a681cbf748, entries=150, sequenceid=252, filesize=11.8 K 2023-03-31 16:54:24,675 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for e76385ddbd37af190ba2df92b5602e3b in 119ms, sequenceid=252, compaction requested=false 2023-03-31 16:54:24,675 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e76385ddbd37af190ba2df92b5602e3b: 2023-03-31 16:54:24,675 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:24,675 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:24,675 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:24,675 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,676 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:24,676 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:24,677 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,678 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:24,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:24,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:24,678 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:24,678 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:24,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:24,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:24,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:24,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:24,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:24,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2da59272[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:24,680 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:24,680 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:24,681 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:24,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:24,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:24,681 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:24,681 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:24,681 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:24,683 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:24,683 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:24,683 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:24,683 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,683 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:24,683 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:24,683 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:24,683 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:24,683 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:24,683 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:24,683 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:24,686 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:24,686 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:24,686 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:24,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:24,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:24,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:24,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:24,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:24,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:24,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:24,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:24,687 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:24,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:24,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:24,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:24,689 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:24,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:24,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,693 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,693 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:24,693 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:24,693 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,693 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:24,693 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:24,693 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:24,693 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:24,693 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:24,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:24,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:24,693 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:24,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:24,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:24,694 DEBUG [Thread-907] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:24,694 DEBUG [Thread-907] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:24,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:24,694 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:24,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:24,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on e76385ddbd37af190ba2df92b5602e3b 2023-03-31 16:54:24,715 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e76385ddbd37af190ba2df92b5602e3b 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-31 16:54:24,715 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e76385ddbd37af190ba2df92b5602e3b, store=A 2023-03-31 16:54:24,716 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:24,716 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e76385ddbd37af190ba2df92b5602e3b, store=B 2023-03-31 16:54:24,716 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:24,716 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e76385ddbd37af190ba2df92b5602e3b, store=C 2023-03-31 16:54:24,716 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:24,738 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:24,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52920 deadline: 1680281724735, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,741 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023033152562d917ff44df2aea9126dcdc4f6f4 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033152562d917ff44df2aea9126dcdc4f6f4 2023-03-31 16:54:24,744 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:24,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52878 deadline: 1680281724737, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,744 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:24,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52892 deadline: 1680281724737, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,744 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:24,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52872 deadline: 1680281724738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,794 DEBUG [Thread-907] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:24,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:24,797 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:24,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:24,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:24,799 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:24,799 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:24,799 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:24,799 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:24,800 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:24,800 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:24,802 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:24,802 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:24,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:24,803 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,803 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:24,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:24,803 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:24,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:24,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:24,804 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:24,808 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:24,808 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:24,808 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:24,808 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:24,808 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:24,808 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:24,808 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:24,808 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:24,808 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. started... 2023-03-31 16:54:24,808 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. as already flushing 2023-03-31 16:54:24,808 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:24,809 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:24,809 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:24,809 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,809 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:24,809 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:24,811 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,811 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:24,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:24,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:24,811 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:24,811 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:24,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:24,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:24,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:24,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:24,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:24,814 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f927b2f[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:24,814 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:24,814 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:24,815 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:24,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:24,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:24,815 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:24,815 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:24,815 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:24,815 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,815 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:24,816 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,817 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:24,817 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:24,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:24,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:24,817 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:24,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:24,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:24,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:24,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:24,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:24,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:24,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:24,822 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:24,822 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:24,822 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:24,822 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:24,822 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:24,828 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:24,828 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:24,828 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:24,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:24,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:24,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:24,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:24,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:24,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:24,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:24,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:24,829 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:24,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:24,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:24,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:24,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,831 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:24,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:24,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,836 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:24,836 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:24,836 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:24,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:24,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:24,836 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:24,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:24,836 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:24,836 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:24,836 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,836 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:24,836 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:24,836 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:24,836 DEBUG [Thread-907] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:24,836 DEBUG [Thread-907] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:24,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:24,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:24,840 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:24,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52920 deadline: 1680281724839, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,850 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:24,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52872 deadline: 1680281724845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,851 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:24,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52892 deadline: 1680281724846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,851 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:24,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52878 deadline: 1680281724846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,927 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/A/c34ee73e86f74737846647f622780292 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/c34ee73e86f74737846647f622780292 2023-03-31 16:54:24,932 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in e76385ddbd37af190ba2df92b5602e3b/A of e76385ddbd37af190ba2df92b5602e3b into c34ee73e86f74737846647f622780292(size=25.7 K), total size for store is 51.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:54:24,932 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e76385ddbd37af190ba2df92b5602e3b: 2023-03-31 16:54:24,932 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b., storeName=e76385ddbd37af190ba2df92b5602e3b/A, priority=10, startTime=1680281664422; duration=0sec 2023-03-31 16:54:24,933 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:54:24,937 DEBUG [Thread-907] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:24,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:24,939 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:24,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:24,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:24,944 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:24,944 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:24,944 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:24,944 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:24,945 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:24,945 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:24,946 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,946 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:24,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:24,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:24,947 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,947 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:24,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:24,947 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:24,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:24,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:24,947 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:24,948 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:24,948 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:24,948 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:24,948 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:24,948 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:24,948 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:24,948 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:24,948 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:24,948 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. started... 2023-03-31 16:54:24,948 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. as already flushing 2023-03-31 16:54:24,948 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:24,948 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:24,948 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:24,948 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,948 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:24,948 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:24,953 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:24,953 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:24,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:24,954 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:24,954 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:24,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:24,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:24,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:24,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:24,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:24,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6bbdcf6e[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:24,955 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:24,955 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:24,957 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:24,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:24,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:24,957 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:24,957 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:24,957 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,957 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:24,957 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:24,958 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:24,958 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,958 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:24,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:24,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:24,958 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:24,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:24,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:24,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:24,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:24,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:24,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:24,961 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:24,961 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:24,961 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:24,961 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:24,961 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:24,961 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:24,962 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:24,962 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:24,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:24,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:24,963 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:24,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:24,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:24,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:24,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:24,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:24,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:24,963 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:24,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:24,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:24,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:24,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,965 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:24,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:24,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,968 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,968 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:24,968 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:24,968 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:24,968 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:24,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:24,968 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:24,968 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:24,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:24,968 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:24,968 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:24,969 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:24,968 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:24,969 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:24,969 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:24,969 DEBUG [Thread-907] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:24,969 DEBUG [Thread-907] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:24,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:24,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:25,042 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:25,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52920 deadline: 1680281725042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,052 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:25,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52872 deadline: 1680281725051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,052 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:25,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52892 deadline: 1680281725052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,055 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:25,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52878 deadline: 1680281725053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,069 DEBUG [Thread-907] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:25,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:25,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:25,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:25,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:25,072 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:25,072 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:25,072 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:25,072 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:25,073 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:25,073 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:25,074 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,074 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:25,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:25,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:25,074 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,074 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:25,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:25,075 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:25,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:25,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:25,075 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:25,075 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:25,075 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:25,075 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:25,075 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:25,075 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:25,076 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:25,076 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:25,076 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:25,076 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. started... 2023-03-31 16:54:25,076 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. as already flushing 2023-03-31 16:54:25,076 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:25,076 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:25,076 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:25,076 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,076 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:25,076 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:25,077 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,077 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:25,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:25,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:25,078 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:25,078 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:25,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:25,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:25,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:25,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:25,079 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:25,079 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:25,079 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63e5a755[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:25,079 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:25,080 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:25,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:25,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:25,080 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:25,081 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:25,081 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:25,081 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,081 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:25,083 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:25,083 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,083 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:25,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:25,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:25,083 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:25,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:25,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:25,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:25,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:25,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:25,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:25,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:25,086 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:25,086 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:25,086 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:25,086 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:25,086 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:25,087 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:25,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:25,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:25,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:25,087 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:25,088 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:25,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:25,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:25,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:25,088 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:25,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:25,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:25,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:25,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:25,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:25,090 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:25,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:25,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,093 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:25,093 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:25,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:25,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:25,093 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,093 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:25,093 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:25,093 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:25,094 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:25,094 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:25,093 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,094 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:25,094 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:25,094 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:25,094 DEBUG [Thread-907] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:25,094 DEBUG [Thread-907] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:25,094 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:25,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:25,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:25,149 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=264, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/A/a14f29985ffd48d4b519bb35864b2493 2023-03-31 16:54:25,194 DEBUG [Thread-907] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:25,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:25,196 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:25,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:25,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:25,197 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:25,197 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:25,197 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:25,197 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:25,200 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:25,201 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:25,202 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:25,202 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:25,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:25,202 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,202 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:25,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:25,202 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:25,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:25,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:25,203 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:25,203 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:25,203 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:25,203 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:25,203 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:25,203 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:25,203 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:25,204 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:25,204 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:25,204 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. started... 2023-03-31 16:54:25,204 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. as already flushing 2023-03-31 16:54:25,204 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:25,204 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:25,204 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:25,204 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,204 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:25,204 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:25,206 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,206 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:25,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:25,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:25,206 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:25,206 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:25,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:25,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:25,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:25,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:25,208 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:25,208 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62d29f59[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:25,208 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:25,208 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:25,209 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:25,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:25,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:25,209 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:25,209 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:25,209 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:25,209 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,209 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:25,210 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,210 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:25,210 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:25,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:25,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:25,210 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:25,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:25,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:25,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:25,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:25,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:25,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:25,213 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:25,213 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:25,213 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:25,213 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:25,213 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:25,213 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:25,214 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:25,214 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:25,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:25,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:25,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:25,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:25,214 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:25,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:25,215 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:25,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:25,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:25,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:25,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:25,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:25,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:25,216 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:25,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:25,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,220 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:25,220 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:25,220 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:25,220 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:25,220 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:25,220 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:25,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:25,220 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:25,221 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:25,221 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,221 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:25,221 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:25,221 DEBUG [Thread-907] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:25,221 DEBUG [Thread-907] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:25,221 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:25,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:25,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:25,321 DEBUG [Thread-907] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:25,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:25,323 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:25,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:25,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:25,325 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:25,325 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:25,325 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:25,325 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:25,325 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:25,325 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:25,326 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,326 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:25,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:25,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:25,327 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,327 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:25,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:25,327 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:25,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:25,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:25,327 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:25,328 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:25,328 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:25,328 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:25,328 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:25,328 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:25,328 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:25,328 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:25,328 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:25,328 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. started... 2023-03-31 16:54:25,328 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. as already flushing 2023-03-31 16:54:25,328 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:25,328 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:25,328 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:25,328 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,328 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:25,328 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:25,330 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,330 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:25,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:25,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:25,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:25,330 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:25,330 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:25,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:25,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:25,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:25,331 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:25,331 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e75db2f[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:25,331 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:25,331 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:25,339 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:25,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:25,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:25,339 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:25,339 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:25,339 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,339 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:25,339 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:25,340 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,340 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:25,341 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:25,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:25,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:25,341 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:25,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:25,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:25,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:25,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:25,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:25,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:25,343 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:25,343 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:25,343 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:25,343 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:25,343 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:25,343 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:25,344 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:25,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52920 deadline: 1680281725344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,345 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:25,345 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:25,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:25,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:25,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:25,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:25,345 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:25,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:25,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:25,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:25,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:25,346 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:25,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:25,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:25,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:25,348 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:25,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:25,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,350 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,350 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:25,350 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:25,350 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:25,350 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,351 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:25,351 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:25,351 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:25,351 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:25,351 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:25,351 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:25,352 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:25,352 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:25,352 DEBUG [Thread-907] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:25,352 DEBUG [Thread-907] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:25,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:25,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:25,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:25,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:25,354 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:25,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52892 deadline: 1680281725354, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,358 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:25,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52872 deadline: 1680281725355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,358 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:25,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52878 deadline: 1680281725357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,452 DEBUG [Thread-907] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:25,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:25,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:25,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:25,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:25,455 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:25,455 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:25,456 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:25,456 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:25,456 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:25,456 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:25,458 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:25,458 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:25,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:25,459 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,459 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:25,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:25,459 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:25,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:25,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:25,460 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:25,460 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:25,460 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:25,460 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:25,460 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:25,460 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:25,460 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:25,460 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:25,460 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:25,460 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. started... 2023-03-31 16:54:25,460 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. as already flushing 2023-03-31 16:54:25,460 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:25,460 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:25,460 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:25,461 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,461 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:25,461 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:25,462 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:25,462 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:25,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:25,462 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:25,462 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:25,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:25,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:25,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:25,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:25,464 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:25,464 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@bb6b330[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:25,464 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:25,464 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:25,465 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:25,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:25,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:25,465 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:25,465 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:25,465 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:25,465 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,465 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:25,466 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,466 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:25,466 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:25,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:25,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:25,467 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:25,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:25,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:25,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:25,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:25,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:25,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:25,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:25,469 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:25,469 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:25,469 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:25,469 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:25,469 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:25,471 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:25,471 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:25,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:25,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:25,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:25,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:25,472 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:25,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:25,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:25,472 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:25,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:25,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:25,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:25,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:25,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:25,474 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:25,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:25,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,476 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:25,476 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:25,476 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:25,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:25,476 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:25,476 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:25,476 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:25,476 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,476 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:25,476 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:25,476 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:25,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:25,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:25,476 DEBUG [Thread-907] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:25,476 DEBUG [Thread-907] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:25,477 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:25,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:25,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:25,560 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=264 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/B/589a73044ba543aaa88615b5c7fa81ef 2023-03-31 16:54:25,576 DEBUG [Thread-907] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:25,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:25,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:25,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:25,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:25,579 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:25,579 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:25,579 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:25,580 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:25,580 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:25,580 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:25,581 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,581 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:25,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:25,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:25,581 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,581 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:25,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:25,582 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:25,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:25,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:25,582 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:25,582 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:25,582 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:25,582 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:25,582 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:25,582 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:25,583 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:25,583 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:25,583 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:25,583 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. started... 2023-03-31 16:54:25,583 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. as already flushing 2023-03-31 16:54:25,583 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:25,583 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:25,583 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:25,583 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,583 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:25,583 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:25,584 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:25,584 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:25,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:25,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:25,585 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:25,585 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:25,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:25,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:25,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:25,588 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:25,588 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4527721c[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:25,588 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:25,588 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:25,590 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:25,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:25,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:25,590 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:25,590 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,590 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:25,590 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:25,591 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:25,592 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,592 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:25,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:25,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:25,592 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:25,592 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:25,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:25,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:25,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:25,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:25,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:25,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:25,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:25,594 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:25,594 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:25,595 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:25,595 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:25,595 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:25,596 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:25,596 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:25,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:25,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:25,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:25,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:25,596 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:25,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:25,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:25,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:25,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:25,596 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:25,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:25,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:25,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:25,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,601 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:25,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:25,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,603 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:25,603 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:25,604 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,604 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:25,604 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:25,604 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:25,604 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:25,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:25,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:25,604 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:25,604 DEBUG [Thread-907] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:25,604 DEBUG [Thread-907] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:25,604 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,604 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:25,604 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:25,604 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:25,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:25,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:25,704 DEBUG [Thread-907] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:25,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:25,706 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:25,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:25,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:25,707 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:25,707 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:25,707 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:25,707 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:25,708 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:25,708 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:25,710 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:25,710 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:25,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:25,711 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,711 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:25,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:25,711 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:25,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:25,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:25,711 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:25,712 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:25,712 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:25,712 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:25,712 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:25,712 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:25,712 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:25,712 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:25,712 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:25,713 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. started... 2023-03-31 16:54:25,713 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. as already flushing 2023-03-31 16:54:25,713 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:25,713 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:25,713 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:25,713 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,713 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:25,713 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:25,714 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:25,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:25,714 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:25,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:25,715 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:25,715 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:25,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:25,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:25,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:25,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:25,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33f915b6[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:25,716 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:25,716 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:25,717 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:25,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:25,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:25,717 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:25,717 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:25,717 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,717 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:25,717 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:25,719 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,719 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:25,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:25,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:25,719 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:25,719 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:25,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:25,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:25,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:25,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:25,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:25,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:25,722 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:25,722 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:25,722 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:25,722 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:25,722 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:25,722 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:25,726 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:25,726 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:25,726 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:25,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:25,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:25,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:25,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:25,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:25,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:25,727 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:25,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:25,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:25,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:25,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:25,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:25,728 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:25,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:25,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,731 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,731 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:25,731 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:25,731 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,731 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:25,731 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:25,731 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:25,731 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:25,731 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:25,731 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:25,731 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:25,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:25,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:25,731 DEBUG [Thread-907] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:25,731 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:25,731 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:25,731 DEBUG [Thread-907] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:25,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:25,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:25,831 DEBUG [Thread-907] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:25,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:25,834 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:25,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:25,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:25,835 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:25,835 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:25,835 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:25,835 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:25,835 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:25,835 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:25,837 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:25,837 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:25,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:25,837 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,837 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:25,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:25,837 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:25,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:25,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:25,838 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:25,838 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:25,838 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:25,838 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:25,838 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:25,838 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:25,838 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:25,838 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:25,838 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:25,839 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. started... 2023-03-31 16:54:25,839 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. as already flushing 2023-03-31 16:54:25,839 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:25,839 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:25,839 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:25,839 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,839 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:25,839 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:25,840 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,840 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:25,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:25,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:25,841 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:25,841 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:25,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:25,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:25,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:25,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:25,842 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:25,842 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13751c45[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:25,842 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:25,842 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:25,843 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:25,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:25,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:25,843 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:25,843 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:25,843 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:25,843 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,843 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:25,845 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:25,845 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:25,845 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,845 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:25,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:25,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:25,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:25,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:25,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:25,847 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:25,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52920 deadline: 1680281725846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:25,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:25,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:25,848 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:25,848 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:25,848 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:25,848 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:25,848 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:25,848 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:25,850 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:25,850 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:25,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:25,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:25,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:25,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:25,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:25,850 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:25,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:25,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:25,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:25,850 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:25,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:25,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:25,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:25,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,852 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:25,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:25,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,853 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,853 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:25,853 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:25,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:25,854 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:25,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:25,854 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:25,854 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:25,854 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:25,854 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:25,854 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,854 DEBUG [Thread-907] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:25,854 DEBUG [Thread-907] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:25,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:25,854 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:25,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:25,854 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:25,854 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:25,860 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:25,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52872 deadline: 1680281725859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,861 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:25,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52892 deadline: 1680281725859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,861 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:25,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52878 deadline: 1680281725860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,954 DEBUG [Thread-907] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:25,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:25,957 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:25,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:25,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:25,958 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:25,958 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:25,958 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:25,958 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:25,958 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:25,958 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:25,959 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:25,959 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:25,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:25,960 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:25,960 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:25,960 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:25,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:25,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:25,961 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:25,961 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:25,961 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:25,961 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:25,961 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:25,961 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:25,961 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:25,961 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:25,961 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:25,961 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. started... 2023-03-31 16:54:25,961 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. as already flushing 2023-03-31 16:54:25,961 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:25,961 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:25,961 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:25,961 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,961 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:25,961 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:25,963 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:25,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:25,963 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:25,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:25,964 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:25,964 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:25,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:25,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:25,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:25,968 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:25,968 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d640228[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:25,968 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:25,968 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:25,969 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:25,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:25,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:25,970 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:25,970 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:25,970 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:25,970 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,970 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:25,971 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=264 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/C/9bcd24273bb64a08a0a72f382aec503c 2023-03-31 16:54:25,975 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/A/a14f29985ffd48d4b519bb35864b2493 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/a14f29985ffd48d4b519bb35864b2493 2023-03-31 16:54:25,977 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:25,977 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:25,977 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:25,977 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:25,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:25,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:25,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:25,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:25,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,980 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/a14f29985ffd48d4b519bb35864b2493, entries=200, sequenceid=264, filesize=32.3 K 2023-03-31 16:54:25,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:25,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:25,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:25,982 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:25,982 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:25,983 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:25,983 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:25,983 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:25,983 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:25,983 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/B/589a73044ba543aaa88615b5c7fa81ef as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/589a73044ba543aaa88615b5c7fa81ef 2023-03-31 16:54:25,985 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:25,985 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:25,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:25,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:25,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:25,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:25,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:25,985 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:25,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:25,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:25,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:25,986 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:25,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:25,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:25,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:25,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:25,989 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:25,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,990 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/589a73044ba543aaa88615b5c7fa81ef, entries=150, sequenceid=264, filesize=11.9 K 2023-03-31 16:54:25,991 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/C/9bcd24273bb64a08a0a72f382aec503c as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/9bcd24273bb64a08a0a72f382aec503c 2023-03-31 16:54:25,992 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:25,992 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:25,992 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:25,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:25,992 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:25,992 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:25,993 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:25,993 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:25,993 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:25,993 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:25,993 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:25,993 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:25,993 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:25,993 DEBUG [Thread-907] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:25,993 DEBUG [Thread-907] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:25,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:25,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:26,003 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/9bcd24273bb64a08a0a72f382aec503c, entries=150, sequenceid=264, filesize=11.9 K 2023-03-31 16:54:26,004 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for e76385ddbd37af190ba2df92b5602e3b in 1289ms, sequenceid=264, compaction requested=true 2023-03-31 16:54:26,004 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e76385ddbd37af190ba2df92b5602e3b: 2023-03-31 16:54:26,004 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:54:26,004 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 16:54:26,004 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 16:54:26,004 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:54:26,004 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:54:26,005 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36668 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 16:54:26,005 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): e76385ddbd37af190ba2df92b5602e3b/B is initiating minor compaction (all files) 2023-03-31 16:54:26,005 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e76385ddbd37af190ba2df92b5602e3b/B in TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:26,005 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/e0e35469f2904e61bc0de0b657e04362, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/1f136f02b8db48448907ae84cb9a687c, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/589a73044ba543aaa88615b5c7fa81ef] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp, totalSize=35.8 K 2023-03-31 16:54:26,006 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting e0e35469f2904e61bc0de0b657e04362, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1680281663882 2023-03-31 16:54:26,006 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85288 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 16:54:26,006 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1912): e76385ddbd37af190ba2df92b5602e3b/A is initiating minor compaction (all files) 2023-03-31 16:54:26,006 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e76385ddbd37af190ba2df92b5602e3b/A in TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:26,006 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/c34ee73e86f74737846647f622780292, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/08825b5aafa943f19831152a4a83b070, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/a14f29985ffd48d4b519bb35864b2493] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp, totalSize=83.3 K 2023-03-31 16:54:26,006 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 1f136f02b8db48448907ae84cb9a687c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=252, earliestPutTs=1680281663970 2023-03-31 16:54:26,007 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting c34ee73e86f74737846647f622780292, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1680281663882 2023-03-31 16:54:26,007 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 589a73044ba543aaa88615b5c7fa81ef, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=264, earliestPutTs=1680281664606 2023-03-31 16:54:26,007 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 08825b5aafa943f19831152a4a83b070, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=252, earliestPutTs=1680281663970 2023-03-31 16:54:26,008 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting a14f29985ffd48d4b519bb35864b2493, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=264, earliestPutTs=1680281664606 2023-03-31 16:54:26,023 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): e76385ddbd37af190ba2df92b5602e3b#B#compaction#164 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:54:26,064 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] throttle.PressureAwareThroughputController(145): e76385ddbd37af190ba2df92b5602e3b#A#compaction#165 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:54:26,072 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/B/35d780aa28f64ba0a3a70e9aebecebc9 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/35d780aa28f64ba0a3a70e9aebecebc9 2023-03-31 16:54:26,079 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e76385ddbd37af190ba2df92b5602e3b/B of e76385ddbd37af190ba2df92b5602e3b into 35d780aa28f64ba0a3a70e9aebecebc9(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:54:26,079 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e76385ddbd37af190ba2df92b5602e3b: 2023-03-31 16:54:26,079 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b., storeName=e76385ddbd37af190ba2df92b5602e3b/B, priority=13, startTime=1680281666004; duration=0sec 2023-03-31 16:54:26,079 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:54:26,079 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 16:54:26,081 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36668 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 16:54:26,081 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): e76385ddbd37af190ba2df92b5602e3b/C is initiating minor compaction (all files) 2023-03-31 16:54:26,081 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e76385ddbd37af190ba2df92b5602e3b/C in TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:26,081 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/33c0856c380343b488cda9c1d784fe0d, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/34b28e01df4347a3a3f785a681cbf748, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/9bcd24273bb64a08a0a72f382aec503c] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp, totalSize=35.8 K 2023-03-31 16:54:26,082 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 33c0856c380343b488cda9c1d784fe0d, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1680281663882 2023-03-31 16:54:26,083 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 34b28e01df4347a3a3f785a681cbf748, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=252, earliestPutTs=1680281663970 2023-03-31 16:54:26,083 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 9bcd24273bb64a08a0a72f382aec503c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=264, earliestPutTs=1680281664606 2023-03-31 16:54:26,093 DEBUG [Thread-907] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:26,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:26,095 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:26,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:26,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:26,101 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:26,101 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:26,101 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:26,101 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:26,101 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:26,101 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:26,103 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): e76385ddbd37af190ba2df92b5602e3b#C#compaction#166 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:54:26,103 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:26,103 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:26,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:26,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:26,104 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:26,104 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:26,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:26,104 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:26,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:26,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:26,105 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:26,120 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:26,120 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:26,120 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:26,120 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:26,120 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:26,120 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:26,120 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:26,120 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:26,120 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. started... 2023-03-31 16:54:26,121 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e76385ddbd37af190ba2df92b5602e3b 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-31 16:54:26,121 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e76385ddbd37af190ba2df92b5602e3b, store=A 2023-03-31 16:54:26,121 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:26,121 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e76385ddbd37af190ba2df92b5602e3b, store=B 2023-03-31 16:54:26,121 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:26,121 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e76385ddbd37af190ba2df92b5602e3b, store=C 2023-03-31 16:54:26,121 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:26,144 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/C/06da64f690c84b029c942dac32b160b1 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/06da64f690c84b029c942dac32b160b1 2023-03-31 16:54:26,151 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e76385ddbd37af190ba2df92b5602e3b/C of e76385ddbd37af190ba2df92b5602e3b into 06da64f690c84b029c942dac32b160b1(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:54:26,151 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e76385ddbd37af190ba2df92b5602e3b: 2023-03-31 16:54:26,151 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b., storeName=e76385ddbd37af190ba2df92b5602e3b/C, priority=13, startTime=1680281666004; duration=0sec 2023-03-31 16:54:26,151 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:54:26,515 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/A/1b92ac5c3d7b4befad2f12095f9ef1e1 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/1b92ac5c3d7b4befad2f12095f9ef1e1 2023-03-31 16:54:26,520 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e76385ddbd37af190ba2df92b5602e3b/A of e76385ddbd37af190ba2df92b5602e3b into 1b92ac5c3d7b4befad2f12095f9ef1e1(size=25.9 K), total size for store is 25.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:54:26,520 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e76385ddbd37af190ba2df92b5602e3b: 2023-03-31 16:54:26,520 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b., storeName=e76385ddbd37af190ba2df92b5602e3b/A, priority=13, startTime=1680281666004; duration=0sec 2023-03-31 16:54:26,520 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:54:26,547 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303319656a47ed53a47d5a0771c4c2ea4ce3b to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303319656a47ed53a47d5a0771c4c2ea4ce3b 2023-03-31 16:54:26,554 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=286, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/A/47141897df044eca95b05aed894c5d18 2023-03-31 16:54:26,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on e76385ddbd37af190ba2df92b5602e3b 2023-03-31 16:54:26,851 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. as already flushing 2023-03-31 16:54:26,866 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:26,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52878 deadline: 1680281726865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:26,867 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:26,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52920 deadline: 1680281726865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:26,867 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:26,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52892 deadline: 1680281726867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:26,870 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:26,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52872 deadline: 1680281726869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:26,968 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:26,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52892 deadline: 1680281726968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:26,969 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:26,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52920 deadline: 1680281726968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:26,982 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=286 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/B/d852aa887d204a75a491acf54ea7df30 2023-03-31 16:54:27,008 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=286 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/C/efe3fbbd80934e7e896de573c8cf8ddf 2023-03-31 16:54:27,014 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/A/47141897df044eca95b05aed894c5d18 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/47141897df044eca95b05aed894c5d18 2023-03-31 16:54:27,019 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/47141897df044eca95b05aed894c5d18, entries=150, sequenceid=286, filesize=25.4 K 2023-03-31 16:54:27,020 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/B/d852aa887d204a75a491acf54ea7df30 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/d852aa887d204a75a491acf54ea7df30 2023-03-31 16:54:27,025 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/d852aa887d204a75a491acf54ea7df30, entries=150, sequenceid=286, filesize=11.9 K 2023-03-31 16:54:27,032 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/C/efe3fbbd80934e7e896de573c8cf8ddf as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/efe3fbbd80934e7e896de573c8cf8ddf 2023-03-31 16:54:27,038 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/efe3fbbd80934e7e896de573c8cf8ddf, entries=150, sequenceid=286, filesize=11.9 K 2023-03-31 16:54:27,039 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for e76385ddbd37af190ba2df92b5602e3b in 918ms, sequenceid=286, compaction requested=false 2023-03-31 16:54:27,039 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e76385ddbd37af190ba2df92b5602e3b: 2023-03-31 16:54:27,039 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:27,039 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:27,039 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:27,039 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,039 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:27,039 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:27,042 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:27,042 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:27,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:27,042 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:27,042 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:27,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:27,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:27,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:27,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:27,045 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:27,045 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@532d220a[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:27,045 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:27,045 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:27,046 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:27,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:27,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:27,047 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:27,047 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:27,047 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:27,047 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,047 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:27,048 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,048 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:27,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:27,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:27,048 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:27,048 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:27,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:27,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:27,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:27,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:27,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:27,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:27,052 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:27,052 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:27,052 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:27,052 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:27,052 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:27,052 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:27,054 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:27,054 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:27,054 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:27,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:27,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:27,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:27,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:27,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:27,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:27,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:27,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:27,055 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:27,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:27,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:27,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:27,057 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:27,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:27,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,064 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:27,064 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:27,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:27,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:27,064 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:27,064 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:27,064 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:27,064 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:27,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:27,064 DEBUG [Thread-907] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:27,064 DEBUG [Thread-907] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:27,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:27,065 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,065 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:27,065 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:27,065 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,065 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:27,065 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:27,065 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:27,164 DEBUG [Thread-907] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:27,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:27,167 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:27,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:27,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:27,168 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:27,168 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:27,168 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:27,168 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:27,170 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:27,170 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:27,172 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e76385ddbd37af190ba2df92b5602e3b 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-31 16:54:27,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on e76385ddbd37af190ba2df92b5602e3b 2023-03-31 16:54:27,172 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,172 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:27,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:27,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:27,173 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e76385ddbd37af190ba2df92b5602e3b, store=A 2023-03-31 16:54:27,173 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:27,173 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e76385ddbd37af190ba2df92b5602e3b, store=B 2023-03-31 16:54:27,173 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:27,173 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e76385ddbd37af190ba2df92b5602e3b, store=C 2023-03-31 16:54:27,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:27,173 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:27,173 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,173 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:27,174 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:27,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:27,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:27,174 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:27,177 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:27,177 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:27,177 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:27,177 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:27,177 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:27,177 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:27,177 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:27,177 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:27,178 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. started... 2023-03-31 16:54:27,178 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. as already flushing 2023-03-31 16:54:27,178 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:27,178 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:27,178 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:27,178 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,178 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:27,178 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:27,179 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:27,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:27,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:27,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:27,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:27,181 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:27,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,182 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:27,182 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:27,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:27,183 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:27,183 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2882c97c[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:27,183 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:27,183 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:27,184 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:27,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:27,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:27,184 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:27,184 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:27,184 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:27,184 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,184 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:27,185 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:27,185 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:27,186 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:27,186 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:27,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:27,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:27,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:27,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:27,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:27,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:27,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:27,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:27,188 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:27,188 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:27,188 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:27,188 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:27,188 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:27,192 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:27,192 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:27,192 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:27,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:27,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:27,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:27,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:27,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:27,193 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:27,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:27,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:27,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:27,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:27,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:27,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:27,194 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:27,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:27,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,196 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,196 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:27,196 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:27,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:27,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:27,196 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:27,197 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:27,197 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,197 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:27,196 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:27,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:27,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:27,196 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:27,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:27,197 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:27,197 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:27,197 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:27,197 DEBUG [Thread-907] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:27,197 DEBUG [Thread-907] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:27,202 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:27,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52920 deadline: 1680281727200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,204 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:27,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52892 deadline: 1680281727200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,298 DEBUG [Thread-907] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:27,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:27,300 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:27,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:27,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:27,301 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:27,301 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:27,301 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:27,301 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:27,301 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:27,301 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:27,303 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,303 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:27,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:27,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:27,303 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,303 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:27,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:27,303 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:27,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:27,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:27,304 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:27,304 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:27,304 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:27,304 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:27,304 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:27,304 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:27,304 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:27,305 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:27,305 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:27,305 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. started... 2023-03-31 16:54:27,305 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. as already flushing 2023-03-31 16:54:27,305 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:27,305 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:27,305 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:27,305 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,305 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:27,305 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:27,305 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:27,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52920 deadline: 1680281727303, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,306 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:27,306 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:27,306 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52892 deadline: 1680281727305, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:27,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:27,307 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:27,307 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:27,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:27,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:27,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:27,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:27,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:27,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13ae6f1d[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:27,308 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:27,308 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:27,311 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:27,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:27,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:27,311 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:27,311 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:27,311 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:27,312 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,312 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:27,312 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:27,313 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:27,312 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,313 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:27,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:27,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:27,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:27,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:27,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:27,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:27,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:27,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:27,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:27,317 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:27,317 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:27,317 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:27,317 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:27,317 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:27,319 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:27,319 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:27,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:27,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:27,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:27,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:27,319 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:27,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:27,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:27,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:27,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:27,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:27,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:27,320 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:27,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:27,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:27,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:27,322 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,324 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,324 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:27,324 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:27,324 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:27,324 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:27,324 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,324 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:27,324 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:27,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:27,324 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:27,324 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:27,324 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:27,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:27,324 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:27,325 DEBUG [Thread-907] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:27,325 DEBUG [Thread-907] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:27,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:27,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:27,425 DEBUG [Thread-907] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:27,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:27,427 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:27,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:27,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:27,428 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:27,428 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:27,428 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:27,428 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:27,429 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:27,429 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:27,431 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:27,431 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:27,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:27,432 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,432 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:27,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:27,432 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:27,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:27,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:27,433 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:27,433 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:27,433 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:27,433 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:27,433 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:27,433 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:27,433 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:27,433 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:27,433 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:27,433 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. started... 2023-03-31 16:54:27,433 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. as already flushing 2023-03-31 16:54:27,434 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:27,434 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:27,434 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:27,434 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,434 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:27,434 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:27,435 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:27,435 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:27,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:27,435 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:27,435 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:27,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:27,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:27,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:27,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:27,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:27,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ed64a02[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:27,437 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:27,437 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:27,438 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:27,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:27,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:27,438 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:27,438 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:27,438 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:27,438 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,438 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:27,439 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:27,440 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,440 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:27,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:27,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:27,440 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:27,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:27,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:27,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:27,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:27,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:27,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:27,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:27,443 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:27,443 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:27,443 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:27,443 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:27,443 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:27,445 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:27,445 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:27,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:27,445 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:27,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:27,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:27,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:27,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:27,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:27,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:27,446 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:27,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:27,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:27,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:27,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:27,447 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:27,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:27,454 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:27,454 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,454 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:27,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:27,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:27,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:27,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:27,454 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:27,454 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:27,454 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:27,454 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,454 DEBUG [Thread-907] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:27,454 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:27,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:27,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:27,454 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:27,455 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:27,455 DEBUG [Thread-907] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:27,455 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:27,507 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:27,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52920 deadline: 1680281727507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,509 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:27,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52892 deadline: 1680281727508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,555 DEBUG [Thread-907] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:27,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:27,557 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:27,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:27,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:27,558 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:27,558 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:27,558 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:27,558 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:27,559 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:27,559 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:27,560 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:27,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:27,560 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:27,561 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,561 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:27,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:27,561 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:27,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:27,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:27,562 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:27,562 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:27,562 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:27,562 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:27,562 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:27,562 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:27,575 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:27,575 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:27,575 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:27,575 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. started... 2023-03-31 16:54:27,575 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. as already flushing 2023-03-31 16:54:27,575 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:27,575 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:27,575 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:27,575 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,576 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:27,576 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:27,577 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:27,577 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:27,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:27,578 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:27,578 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:27,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:27,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:27,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:27,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:27,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:27,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@245116a0[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:27,580 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:27,580 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:27,582 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:27,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:27,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:27,582 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:27,582 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:27,582 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:27,583 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,583 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:27,584 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:27,584 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:27,584 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:27,584 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:27,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:27,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:27,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:27,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:27,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:27,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:27,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:27,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:27,586 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:27,586 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:27,586 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:27,587 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:27,587 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:27,588 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:27,588 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:27,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:27,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:27,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:27,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:27,588 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:27,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:27,588 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:27,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:27,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:27,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:27,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:27,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:27,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:27,589 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:27,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:27,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,592 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,592 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:27,592 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:27,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:27,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:27,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:27,592 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:27,592 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:27,592 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:27,592 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:27,592 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:27,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:27,592 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:27,593 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:27,593 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:27,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:27,593 DEBUG [Thread-907] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:27,593 DEBUG [Thread-907] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:27,594 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230331f68a2a0c90244d46aa364f2760dbbb09 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331f68a2a0c90244d46aa364f2760dbbb09 2023-03-31 16:54:27,693 DEBUG [Thread-907] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:27,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:27,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:27,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:27,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:27,696 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:27,696 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:27,696 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:27,696 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:27,697 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:27,697 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:27,698 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:27,698 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:27,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:27,698 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,698 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:27,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:27,698 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:27,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:27,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:27,699 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:27,699 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:27,699 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:27,699 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:27,699 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:27,699 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:27,700 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:27,700 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:27,700 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:27,700 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. started... 2023-03-31 16:54:27,700 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. as already flushing 2023-03-31 16:54:27,700 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:27,700 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:27,700 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:27,700 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,700 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:27,700 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:27,701 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,701 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:27,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:27,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:27,702 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:27,702 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:27,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:27,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:27,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:27,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:27,703 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:27,704 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:27,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1aa6f27f[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:27,704 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:27,707 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:27,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:27,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:27,707 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:27,707 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:27,707 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:27,707 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,707 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:27,710 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,710 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:27,710 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:27,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:27,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:27,710 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:27,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:27,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:27,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:27,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:27,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:27,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:27,712 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:27,712 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:27,712 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:27,712 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:27,712 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:27,712 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:27,714 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:27,714 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:27,714 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:27,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:27,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:27,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:27,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:27,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:27,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:27,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:27,714 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:27,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:27,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:27,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:27,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:27,715 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:27,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:27,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,719 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,719 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:27,719 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:27,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:27,719 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:27,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:27,719 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:27,719 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:27,719 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:27,719 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:27,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:27,719 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:27,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:27,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:27,720 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:27,720 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:27,720 DEBUG [Thread-907] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:27,720 DEBUG [Thread-907] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:27,811 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:27,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52920 deadline: 1680281727810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,812 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:27,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52892 deadline: 1680281727812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,820 DEBUG [Thread-907] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:27,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:27,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:27,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:27,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:27,823 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:27,823 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:27,823 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:27,823 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:27,824 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:27,824 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:27,826 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:27,826 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:27,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:27,826 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,826 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:27,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:27,826 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:27,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:27,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:27,827 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:27,828 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:27,828 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:27,828 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:27,828 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:27,828 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:27,828 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:27,828 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:27,828 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:27,828 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. started... 2023-03-31 16:54:27,828 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. as already flushing 2023-03-31 16:54:27,828 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:27,828 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:27,828 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:27,828 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,829 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:27,829 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:27,830 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,830 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:27,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:27,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:27,830 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:27,830 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:27,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:27,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:27,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:27,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:27,831 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:27,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a90a0b1[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:27,832 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:27,832 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:27,833 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:27,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:27,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:27,833 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:27,833 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:27,833 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:27,833 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,833 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:27,834 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,834 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:27,834 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:27,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:27,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:27,834 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:27,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:27,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:27,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:27,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:27,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:27,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:27,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:27,836 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:27,837 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:27,837 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:27,837 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:27,837 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:27,839 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:27,839 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:27,839 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:27,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:27,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:27,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:27,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:27,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:27,839 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:27,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:27,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:27,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:27,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:27,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:27,842 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:27,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:27,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:27,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,844 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:27,844 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:27,844 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:27,844 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:27,845 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:27,844 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:27,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:27,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:27,845 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:27,845 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:27,845 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:27,845 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:27,845 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:27,845 DEBUG [Thread-907] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:27,845 DEBUG [Thread-907] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:27,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:27,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:27,945 DEBUG [Thread-907] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:27,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:27,947 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:27,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:27,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:27,949 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:27,949 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:27,949 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:27,949 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:27,949 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:27,949 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:27,950 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:27,950 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:27,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:27,951 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,951 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:27,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:27,951 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:27,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:27,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:27,951 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:27,952 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:27,952 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:27,952 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:27,952 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:27,952 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:27,952 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:27,952 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:27,952 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:27,952 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. started... 2023-03-31 16:54:27,952 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. as already flushing 2023-03-31 16:54:27,953 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:27,953 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:27,953 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:27,953 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,953 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:27,953 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:27,958 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:27,958 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:27,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:27,959 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:27,959 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:27,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:27,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:27,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:27,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:27,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:27,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@563a63bb[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:27,960 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:27,960 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:27,961 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:27,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:27,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:27,961 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:27,961 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:27,961 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:27,962 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,962 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:27,963 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:27,963 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:27,963 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,963 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:27,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:27,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:27,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:27,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:27,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:27,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:27,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:27,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:27,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:27,966 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:27,966 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:27,966 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:27,966 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:27,966 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:27,968 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:27,968 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:27,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:27,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:27,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:27,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:27,968 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:27,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:27,969 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:27,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:27,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:27,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:27,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:27,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:27,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:27,970 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:27,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:27,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,973 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,973 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:27,974 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:27,974 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:27,974 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:27,974 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:27,974 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:27,974 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:27,974 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:27,974 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:27,974 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:27,974 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:27,974 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:27,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:27,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:27,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:27,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:27,975 DEBUG [Thread-907] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:27,975 DEBUG [Thread-907] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:28,001 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=302, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/A/2500ad05fb71487bb308e2b2a9f60c4c 2023-03-31 16:54:28,075 DEBUG [Thread-907] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:28,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:28,077 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:28,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:28,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:28,078 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:28,078 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:28,078 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:28,078 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:28,078 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:28,078 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:28,080 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:28,080 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:28,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:28,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:28,080 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:28,080 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:28,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:28,081 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:28,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:28,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:28,081 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:28,082 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:28,082 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:28,082 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:28,082 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:28,082 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:28,084 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:28,084 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:28,084 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:28,084 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. started... 2023-03-31 16:54:28,084 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. as already flushing 2023-03-31 16:54:28,084 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:28,084 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:28,084 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:28,084 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:28,084 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:28,084 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:28,086 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:28,086 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:28,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:28,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:28,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:28,086 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:28,086 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:28,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:28,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:28,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:28,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:28,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:28,088 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:28,088 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a728bca[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:28,088 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:28,088 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:28,089 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:28,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:28,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:28,089 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:28,089 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:28,089 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:28,090 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:28,090 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:28,091 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:28,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:28,092 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:28,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:28,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:28,092 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:28,092 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:28,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:28,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:28,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:28,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:28,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:28,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:28,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:28,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:28,095 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:28,095 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:28,095 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:28,095 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:28,095 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:28,095 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:28,097 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:28,097 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:28,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:28,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:28,097 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:28,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:28,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:28,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:28,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:28,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:28,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:28,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:28,098 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:28,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:28,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:28,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:28,099 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:28,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:28,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:28,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:28,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:28,101 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:28,101 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:28,101 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:28,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:28,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:28,101 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:28,101 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:28,101 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:28,101 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:28,102 DEBUG [Thread-907] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:28,101 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:28,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:28,102 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:28,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:28,102 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:28,102 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:28,102 DEBUG [Thread-907] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:28,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:28,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:28,202 DEBUG [Thread-907] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:28,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:28,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:28,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:28,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:28,212 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:28,212 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:28,212 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:28,212 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:28,212 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:28,213 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:28,214 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:28,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:28,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:28,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:28,214 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:28,215 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:28,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:28,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:28,215 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:28,216 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:28,216 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:28,216 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:28,216 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:28,216 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:28,216 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:28,216 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:28,216 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:28,216 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:28,217 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:28,217 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. started... 2023-03-31 16:54:28,217 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. as already flushing 2023-03-31 16:54:28,217 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:28,217 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:28,217 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:28,217 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:28,217 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:28,217 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:28,220 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:28,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:28,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:28,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:28,221 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:28,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:28,222 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:28,222 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:28,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:28,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:28,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:28,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:28,223 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:28,223 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@576abbf1[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:28,223 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:28,223 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:28,224 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:28,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:28,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:28,225 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:28,225 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:28,225 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:28,226 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:28,226 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:28,226 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:28,227 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:28,227 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:28,227 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:28,227 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:28,227 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:28,227 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:28,227 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:28,228 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:28,229 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:28,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:28,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:28,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:28,230 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:28,230 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:28,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:28,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:28,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:28,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:28,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:28,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:28,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:28,231 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:28,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:28,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:28,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:28,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:28,233 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:28,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:28,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:28,236 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:28,236 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:28,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:28,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:28,236 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:28,236 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:28,236 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:28,236 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:28,236 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:28,236 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:28,236 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:28,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:28,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:28,240 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:28,240 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:28,240 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:28,240 DEBUG [Thread-907] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:28,240 DEBUG [Thread-907] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:28,315 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:28,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52892 deadline: 1680281728315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:28,316 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:28,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52920 deadline: 1680281728315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:28,340 DEBUG [Thread-907] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:28,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:28,344 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:28,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:28,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:28,345 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:28,345 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:28,345 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:28,345 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:28,345 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:28,345 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:28,347 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:28,347 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:28,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:28,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:28,347 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:28,347 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:28,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:28,348 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:28,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:28,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:28,348 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:28,348 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:28,348 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:28,348 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:28,348 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:28,348 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:28,348 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:28,348 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:28,348 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:28,349 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. started... 2023-03-31 16:54:28,349 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. as already flushing 2023-03-31 16:54:28,349 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:28,349 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:28,349 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:28,349 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:28,349 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:28,349 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:28,350 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:28,351 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:28,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:28,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:28,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:28,351 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:28,351 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:28,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:28,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:28,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:28,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:28,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:28,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:28,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28800c85[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:28,354 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:28,354 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:28,360 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:28,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:28,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:28,360 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:28,360 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:28,360 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:28,360 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:28,360 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:28,364 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:28,364 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:28,364 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:28,365 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:28,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:28,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:28,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:28,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:28,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:28,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:28,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:28,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:28,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:28,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:28,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:28,368 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:28,368 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:28,368 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:28,368 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:28,368 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:28,368 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:28,370 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:28,370 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:28,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:28,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:28,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:28,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:28,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:28,370 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:28,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:28,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:28,370 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:28,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:28,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:28,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:28,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:28,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:28,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:28,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:28,375 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:28,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:28,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:28,379 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:28,379 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:28,379 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:28,379 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:28,379 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:28,379 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:28,379 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:28,380 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:28,380 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:28,380 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:28,380 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:28,380 DEBUG [Thread-907] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:28,379 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:28,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:28,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:28,380 DEBUG [Thread-907] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:28,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:28,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:28,418 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=302 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/B/e0e6d189e2bf4a73bf565a69072640ae 2023-03-31 16:54:28,466 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=302 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/C/a96fa11ba27e4438af1889b2414ac68c 2023-03-31 16:54:28,472 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/A/2500ad05fb71487bb308e2b2a9f60c4c as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/2500ad05fb71487bb308e2b2a9f60c4c 2023-03-31 16:54:28,478 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/2500ad05fb71487bb308e2b2a9f60c4c, entries=200, sequenceid=302, filesize=32.3 K 2023-03-31 16:54:28,480 DEBUG [Thread-907] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:28,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:28,484 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/B/e0e6d189e2bf4a73bf565a69072640ae as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/e0e6d189e2bf4a73bf565a69072640ae 2023-03-31 16:54:28,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:28,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:28,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:28,487 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:28,487 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:28,487 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:28,487 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:28,488 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:28,488 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:28,497 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:28,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:28,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:28,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:28,498 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:28,498 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/e0e6d189e2bf4a73bf565a69072640ae, entries=150, sequenceid=302, filesize=11.9 K 2023-03-31 16:54:28,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:28,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:28,498 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:28,498 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:28,499 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:28,499 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:28,500 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/C/a96fa11ba27e4438af1889b2414ac68c as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/a96fa11ba27e4438af1889b2414ac68c 2023-03-31 16:54:28,504 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:28,504 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:28,504 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:28,504 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:28,504 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:28,506 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/a96fa11ba27e4438af1889b2414ac68c, entries=150, sequenceid=302, filesize=11.9 K 2023-03-31 16:54:28,507 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for e76385ddbd37af190ba2df92b5602e3b in 1335ms, sequenceid=302, compaction requested=true 2023-03-31 16:54:28,507 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e76385ddbd37af190ba2df92b5602e3b: 2023-03-31 16:54:28,507 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:54:28,507 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 16:54:28,507 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 16:54:28,507 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:54:28,508 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:54:28,509 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37070 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 16:54:28,509 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85690 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 16:54:28,509 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1912): e76385ddbd37af190ba2df92b5602e3b/B is initiating minor compaction (all files) 2023-03-31 16:54:28,509 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): e76385ddbd37af190ba2df92b5602e3b/A is initiating minor compaction (all files) 2023-03-31 16:54:28,509 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e76385ddbd37af190ba2df92b5602e3b/B in TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:28,509 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e76385ddbd37af190ba2df92b5602e3b/A in TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:28,509 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/35d780aa28f64ba0a3a70e9aebecebc9, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/d852aa887d204a75a491acf54ea7df30, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/e0e6d189e2bf4a73bf565a69072640ae] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp, totalSize=36.2 K 2023-03-31 16:54:28,509 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/1b92ac5c3d7b4befad2f12095f9ef1e1, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/47141897df044eca95b05aed894c5d18, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/2500ad05fb71487bb308e2b2a9f60c4c] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp, totalSize=83.7 K 2023-03-31 16:54:28,510 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 35d780aa28f64ba0a3a70e9aebecebc9, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=264, earliestPutTs=1680281664606 2023-03-31 16:54:28,510 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 1b92ac5c3d7b4befad2f12095f9ef1e1, keycount=150, bloomtype=ROW, size=25.9 K, encoding=NONE, compression=NONE, seqNum=264, earliestPutTs=1680281664606 2023-03-31 16:54:28,510 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting d852aa887d204a75a491acf54ea7df30, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=286, earliestPutTs=1680281664733 2023-03-31 16:54:28,510 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 47141897df044eca95b05aed894c5d18, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=286, earliestPutTs=1680281664733 2023-03-31 16:54:28,510 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting e0e6d189e2bf4a73bf565a69072640ae, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=302, earliestPutTs=1680281666864 2023-03-31 16:54:28,510 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 2500ad05fb71487bb308e2b2a9f60c4c, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=302, earliestPutTs=1680281666860 2023-03-31 16:54:28,512 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:28,512 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:28,512 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:28,512 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. started... 2023-03-31 16:54:28,512 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e76385ddbd37af190ba2df92b5602e3b 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-31 16:54:28,513 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e76385ddbd37af190ba2df92b5602e3b, store=A 2023-03-31 16:54:28,513 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:28,513 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e76385ddbd37af190ba2df92b5602e3b, store=B 2023-03-31 16:54:28,513 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:28,513 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e76385ddbd37af190ba2df92b5602e3b, store=C 2023-03-31 16:54:28,513 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:28,522 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] throttle.PressureAwareThroughputController(145): e76385ddbd37af190ba2df92b5602e3b#B#compaction#174 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:54:28,534 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): e76385ddbd37af190ba2df92b5602e3b#A#compaction#173 average throughput is 1.41 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:54:28,570 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/B/f803236f03614c06ac67f933c6df5f04 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/f803236f03614c06ac67f933c6df5f04 2023-03-31 16:54:28,576 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e76385ddbd37af190ba2df92b5602e3b/B of e76385ddbd37af190ba2df92b5602e3b into f803236f03614c06ac67f933c6df5f04(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:54:28,576 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e76385ddbd37af190ba2df92b5602e3b: 2023-03-31 16:54:28,576 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b., storeName=e76385ddbd37af190ba2df92b5602e3b/B, priority=13, startTime=1680281668507; duration=0sec 2023-03-31 16:54:28,576 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:54:28,576 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 16:54:28,577 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37070 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 16:54:28,578 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1912): e76385ddbd37af190ba2df92b5602e3b/C is initiating minor compaction (all files) 2023-03-31 16:54:28,578 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e76385ddbd37af190ba2df92b5602e3b/C in TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:28,578 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/06da64f690c84b029c942dac32b160b1, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/efe3fbbd80934e7e896de573c8cf8ddf, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/a96fa11ba27e4438af1889b2414ac68c] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp, totalSize=36.2 K 2023-03-31 16:54:28,578 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 06da64f690c84b029c942dac32b160b1, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=264, earliestPutTs=1680281664606 2023-03-31 16:54:28,579 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting efe3fbbd80934e7e896de573c8cf8ddf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=286, earliestPutTs=1680281664733 2023-03-31 16:54:28,579 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting a96fa11ba27e4438af1889b2414ac68c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=302, earliestPutTs=1680281666864 2023-03-31 16:54:28,587 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] throttle.PressureAwareThroughputController(145): e76385ddbd37af190ba2df92b5602e3b#C#compaction#176 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:54:28,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on e76385ddbd37af190ba2df92b5602e3b 2023-03-31 16:54:28,883 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. as already flushing 2023-03-31 16:54:28,903 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:28,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52872 deadline: 1680281728900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:28,904 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:28,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52878 deadline: 1680281728901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:28,976 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023033109589f253d734a2887fe7f3f34b294cb to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033109589f253d734a2887fe7f3f34b294cb 2023-03-31 16:54:29,007 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/C/726aa24809b043d6adfb7cd24e6e6817 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/726aa24809b043d6adfb7cd24e6e6817 2023-03-31 16:54:29,009 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:29,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52872 deadline: 1680281729004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,011 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:29,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52878 deadline: 1680281729007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,015 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e76385ddbd37af190ba2df92b5602e3b/C of e76385ddbd37af190ba2df92b5602e3b into 726aa24809b043d6adfb7cd24e6e6817(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:54:29,015 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e76385ddbd37af190ba2df92b5602e3b: 2023-03-31 16:54:29,015 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b., storeName=e76385ddbd37af190ba2df92b5602e3b/C, priority=13, startTime=1680281668508; duration=0sec 2023-03-31 16:54:29,018 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:54:29,026 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=320, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/A/4f4a1afb2eb140d6a452686e00497ecd 2023-03-31 16:54:29,035 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/A/536a348080da40f4b72f1ad98f22c240 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/536a348080da40f4b72f1ad98f22c240 2023-03-31 16:54:29,041 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e76385ddbd37af190ba2df92b5602e3b/A of e76385ddbd37af190ba2df92b5602e3b into 536a348080da40f4b72f1ad98f22c240(size=26.0 K), total size for store is 26.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:54:29,041 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e76385ddbd37af190ba2df92b5602e3b: 2023-03-31 16:54:29,041 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b., storeName=e76385ddbd37af190ba2df92b5602e3b/A, priority=13, startTime=1680281668507; duration=0sec 2023-03-31 16:54:29,041 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:54:29,049 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=320 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/B/0848704199784825a46410dd711b14da 2023-03-31 16:54:29,097 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=320 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/C/fdbd906ffb1e4b64906a7d7da8de7432 2023-03-31 16:54:29,111 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/A/4f4a1afb2eb140d6a452686e00497ecd as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/4f4a1afb2eb140d6a452686e00497ecd 2023-03-31 16:54:29,118 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/4f4a1afb2eb140d6a452686e00497ecd, entries=150, sequenceid=320, filesize=25.4 K 2023-03-31 16:54:29,119 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/B/0848704199784825a46410dd711b14da as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/0848704199784825a46410dd711b14da 2023-03-31 16:54:29,124 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/0848704199784825a46410dd711b14da, entries=150, sequenceid=320, filesize=11.9 K 2023-03-31 16:54:29,125 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/C/fdbd906ffb1e4b64906a7d7da8de7432 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/fdbd906ffb1e4b64906a7d7da8de7432 2023-03-31 16:54:29,131 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/fdbd906ffb1e4b64906a7d7da8de7432, entries=150, sequenceid=320, filesize=11.9 K 2023-03-31 16:54:29,132 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=80.51 KB/82440 for e76385ddbd37af190ba2df92b5602e3b in 620ms, sequenceid=320, compaction requested=false 2023-03-31 16:54:29,132 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e76385ddbd37af190ba2df92b5602e3b: 2023-03-31 16:54:29,132 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:29,132 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:29,133 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:29,133 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,133 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:29,133 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:29,134 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:29,134 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:29,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:29,134 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:29,134 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:29,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:29,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:29,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:29,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:29,136 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:29,136 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2da29595[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:29,136 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:29,136 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:29,137 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:29,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:29,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:29,138 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:29,138 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:29,138 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,138 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:29,138 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:29,140 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:29,140 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:29,140 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,140 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:29,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:29,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:29,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:29,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:29,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:29,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:29,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:29,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:29,145 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:29,145 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:29,145 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:29,145 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:29,145 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:29,145 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:29,147 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:29,147 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:29,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:29,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:29,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:29,147 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:29,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:29,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:29,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:29,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:29,148 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:29,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:29,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:29,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:29,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:29,149 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:29,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:29,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,151 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,151 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:29,151 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:29,151 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:29,151 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:29,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:29,151 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:29,151 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:29,151 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:29,151 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:29,151 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:29,151 DEBUG [Thread-907] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:29,151 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:29,151 DEBUG [Thread-907] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:29,152 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:29,152 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:29,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:29,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:29,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on e76385ddbd37af190ba2df92b5602e3b 2023-03-31 16:54:29,213 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e76385ddbd37af190ba2df92b5602e3b 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-31 16:54:29,213 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e76385ddbd37af190ba2df92b5602e3b, store=A 2023-03-31 16:54:29,213 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:29,213 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e76385ddbd37af190ba2df92b5602e3b, store=B 2023-03-31 16:54:29,213 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:29,213 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e76385ddbd37af190ba2df92b5602e3b, store=C 2023-03-31 16:54:29,213 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:29,246 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230331ac9635d9d82f42ec8f02b43f67424353 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331ac9635d9d82f42ec8f02b43f67424353 2023-03-31 16:54:29,251 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=339, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/A/ffea3aea26b5493a93ddda4a6f984be3 2023-03-31 16:54:29,251 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:29,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52878 deadline: 1680281729247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,252 DEBUG [Thread-907] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:29,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:29,254 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:29,255 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:29,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52872 deadline: 1680281729252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:29,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:29,255 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:29,255 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:29,256 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:29,256 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:29,256 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:29,256 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:29,258 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:29,258 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:29,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:29,258 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,258 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:29,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:29,259 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:29,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:29,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:29,259 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:29,259 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:29,259 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:29,259 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:29,259 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:29,260 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:29,260 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:29,260 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:29,260 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:29,260 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. started... 2023-03-31 16:54:29,260 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. as already flushing 2023-03-31 16:54:29,261 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:29,261 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:29,261 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:29,261 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,265 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:29,265 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:29,268 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,268 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:29,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:29,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:29,268 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:29,268 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:29,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:29,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:29,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:29,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:29,270 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:29,270 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43606730[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:29,270 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:29,270 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:29,271 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:29,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:29,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:29,271 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,271 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:29,271 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:29,271 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:29,271 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:29,273 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:29,273 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,273 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:29,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:29,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:29,273 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:29,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:29,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:29,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:29,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:29,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:29,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:29,276 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:29,276 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:29,276 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:29,276 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:29,276 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:29,276 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:29,278 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:29,278 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:29,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:29,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:29,278 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:29,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:29,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:29,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:29,279 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=339 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/B/9ea82a92ae2b4abca98841eb3b399c24 2023-03-31 16:54:29,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:29,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:29,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:29,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:29,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:29,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:29,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:29,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:29,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:29,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,292 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,293 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,297 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,297 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:29,297 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:29,297 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,297 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:29,297 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:29,297 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:29,297 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:29,297 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:29,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:29,297 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:29,297 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:29,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:29,298 DEBUG [Thread-907] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:29,297 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:29,298 DEBUG [Thread-907] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:29,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:29,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:29,300 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=339 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/C/5af78e50c5ac4422afdd16d5ba4c5543 2023-03-31 16:54:29,307 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/A/ffea3aea26b5493a93ddda4a6f984be3 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/ffea3aea26b5493a93ddda4a6f984be3 2023-03-31 16:54:29,317 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/ffea3aea26b5493a93ddda4a6f984be3, entries=200, sequenceid=339, filesize=32.3 K 2023-03-31 16:54:29,317 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/B/9ea82a92ae2b4abca98841eb3b399c24 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/9ea82a92ae2b4abca98841eb3b399c24 2023-03-31 16:54:29,322 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/9ea82a92ae2b4abca98841eb3b399c24, entries=150, sequenceid=339, filesize=11.9 K 2023-03-31 16:54:29,323 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/C/5af78e50c5ac4422afdd16d5ba4c5543 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/5af78e50c5ac4422afdd16d5ba4c5543 2023-03-31 16:54:29,325 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:29,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52920 deadline: 1680281729322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,327 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:29,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52892 deadline: 1680281729326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,327 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/5af78e50c5ac4422afdd16d5ba4c5543, entries=150, sequenceid=339, filesize=11.9 K 2023-03-31 16:54:29,328 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for e76385ddbd37af190ba2df92b5602e3b in 115ms, sequenceid=339, compaction requested=true 2023-03-31 16:54:29,328 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e76385ddbd37af190ba2df92b5602e3b: 2023-03-31 16:54:29,328 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:54:29,328 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 16:54:29,328 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 16:54:29,328 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:54:29,328 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:54:29,329 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37172 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 16:54:29,330 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85792 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 16:54:29,330 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): e76385ddbd37af190ba2df92b5602e3b/B is initiating minor compaction (all files) 2023-03-31 16:54:29,330 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1912): e76385ddbd37af190ba2df92b5602e3b/A is initiating minor compaction (all files) 2023-03-31 16:54:29,330 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e76385ddbd37af190ba2df92b5602e3b/B in TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:29,330 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e76385ddbd37af190ba2df92b5602e3b/A in TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:29,330 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/f803236f03614c06ac67f933c6df5f04, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/0848704199784825a46410dd711b14da, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/9ea82a92ae2b4abca98841eb3b399c24] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp, totalSize=36.3 K 2023-03-31 16:54:29,330 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/536a348080da40f4b72f1ad98f22c240, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/4f4a1afb2eb140d6a452686e00497ecd, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/ffea3aea26b5493a93ddda4a6f984be3] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp, totalSize=83.8 K 2023-03-31 16:54:29,331 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting f803236f03614c06ac67f933c6df5f04, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=302, earliestPutTs=1680281666864 2023-03-31 16:54:29,331 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 536a348080da40f4b72f1ad98f22c240, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=302, earliestPutTs=1680281666864 2023-03-31 16:54:29,331 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 0848704199784825a46410dd711b14da, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=320, earliestPutTs=1680281667181 2023-03-31 16:54:29,332 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 9ea82a92ae2b4abca98841eb3b399c24, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=339, earliestPutTs=1680281668890 2023-03-31 16:54:29,332 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 4f4a1afb2eb140d6a452686e00497ecd, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=320, earliestPutTs=1680281667181 2023-03-31 16:54:29,333 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting ffea3aea26b5493a93ddda4a6f984be3, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=339, earliestPutTs=1680281668890 2023-03-31 16:54:29,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on e76385ddbd37af190ba2df92b5602e3b 2023-03-31 16:54:29,354 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e76385ddbd37af190ba2df92b5602e3b 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-31 16:54:29,354 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e76385ddbd37af190ba2df92b5602e3b, store=A 2023-03-31 16:54:29,354 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:29,354 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e76385ddbd37af190ba2df92b5602e3b, store=B 2023-03-31 16:54:29,355 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:29,355 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e76385ddbd37af190ba2df92b5602e3b, store=C 2023-03-31 16:54:29,355 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:29,356 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): e76385ddbd37af190ba2df92b5602e3b#B#compaction#182 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:54:29,373 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] throttle.PressureAwareThroughputController(145): e76385ddbd37af190ba2df92b5602e3b#A#compaction#183 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:54:29,388 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:29,388 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:29,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52872 deadline: 1680281729384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52878 deadline: 1680281729383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,398 DEBUG [Thread-907] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:29,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:29,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:29,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:29,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:29,408 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:29,408 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:29,408 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:29,408 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:29,408 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:29,408 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:29,410 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:29,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:29,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:29,410 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:29,410 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,410 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:29,410 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:29,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:29,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:29,411 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:29,412 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:29,412 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:29,413 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:29,413 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:29,413 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:29,416 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:29,416 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:29,416 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:29,416 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. started... 2023-03-31 16:54:29,416 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. as already flushing 2023-03-31 16:54:29,416 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:29,416 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:29,416 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:29,416 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,416 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:29,416 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:29,418 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:29,418 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:29,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:29,418 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:29,418 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:29,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:29,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:29,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:29,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:29,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:29,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41d7f99e[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:29,421 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:29,421 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:29,422 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:29,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:29,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:29,423 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:29,423 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:29,423 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:29,423 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,423 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:29,427 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,427 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:29,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:29,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:29,427 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:29,428 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:29,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:29,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:29,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:29,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:29,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:29,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:29,430 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:29,430 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:29,430 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:29,430 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:29,430 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:29,430 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:29,431 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:29,431 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:29,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:29,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:29,432 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:29,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:29,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:29,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:29,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:29,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:29,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:29,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:29,432 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:29,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:29,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:29,435 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:29,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:29,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,438 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:29,438 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:29,438 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,438 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:29,438 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:29,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:29,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:29,438 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:29,438 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:29,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:29,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:29,438 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,439 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:29,439 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:29,439 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:29,439 DEBUG [Thread-907] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:29,439 DEBUG [Thread-907] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:29,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:29,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:29,492 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:29,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52872 deadline: 1680281729490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,492 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:29,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52878 deadline: 1680281729490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,539 DEBUG [Thread-907] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:29,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:29,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:29,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:29,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:29,543 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:29,543 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:29,543 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:29,543 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:29,545 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:29,545 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:29,547 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:29,547 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:29,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:29,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:29,547 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,547 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:29,548 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:29,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:29,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:29,549 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:29,549 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:29,549 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:29,549 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:29,549 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:29,549 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:29,552 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:29,552 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:29,552 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:29,552 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. started... 2023-03-31 16:54:29,552 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. as already flushing 2023-03-31 16:54:29,552 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:29,552 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:29,552 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:29,552 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,552 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:29,552 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:29,557 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,557 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:29,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:29,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:29,557 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:29,557 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:29,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:29,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:29,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:29,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:29,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:29,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@691b2105[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:29,560 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:29,560 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:29,561 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:29,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:29,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:29,561 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:29,561 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:29,561 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,561 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:29,561 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:29,563 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:29,563 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,563 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:29,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:29,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:29,563 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:29,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:29,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:29,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:29,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:29,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:29,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:29,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:29,569 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:29,569 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:29,569 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:29,569 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:29,569 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:29,570 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:29,570 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:29,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:29,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:29,571 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:29,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:29,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:29,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:29,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:29,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:29,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:29,574 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:29,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:29,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:29,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:29,579 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:29,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:29,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,584 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:29,584 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:29,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:29,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:29,584 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:29,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:29,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:29,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:29,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:29,584 DEBUG [Thread-907] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:29,584 DEBUG [Thread-907] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:29,588 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,588 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:29,588 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:29,588 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,588 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:29,588 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:29,588 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:29,685 DEBUG [Thread-907] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:29,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:29,687 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:29,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:29,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:29,688 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:29,688 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:29,688 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:29,688 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:29,694 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:29,694 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:29,695 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,695 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,695 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:29,695 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:29,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:29,696 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:29,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:29,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52872 deadline: 1680281729694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,696 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:29,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52878 deadline: 1680281729694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:29,696 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:29,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:29,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:29,697 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:29,713 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:29,713 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:29,713 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:29,713 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:29,713 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:29,714 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:29,714 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:29,714 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:29,714 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. started... 2023-03-31 16:54:29,714 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. as already flushing 2023-03-31 16:54:29,714 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:29,714 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:29,714 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:29,714 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,714 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:29,714 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:29,716 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:29,716 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:29,717 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:29,717 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:29,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:29,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:29,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:29,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:29,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:29,721 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:29,721 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@502c0f8e[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:29,722 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:29,722 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:29,723 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:29,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:29,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:29,723 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,723 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:29,723 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:29,723 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:29,723 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:29,725 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:29,725 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:29,725 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:29,725 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:29,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:29,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:29,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:29,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:29,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:29,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:29,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:29,741 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:29,741 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:29,741 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:29,741 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:29,741 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:29,741 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:29,743 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:29,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:29,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:29,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:29,744 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:29,744 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:29,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:29,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:29,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:29,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:29,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:29,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:29,745 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:29,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:29,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:29,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,747 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:29,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:29,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,749 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:29,749 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:29,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:29,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:29,750 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,750 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:29,750 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:29,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:29,750 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:29,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:29,750 DEBUG [Thread-907] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:29,750 DEBUG [Thread-907] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:29,750 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:29,750 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:29,750 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,750 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:29,750 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:29,750 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:29,802 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/B/a0883988b1074fa08b9e8bbbfff1b051 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/a0883988b1074fa08b9e8bbbfff1b051 2023-03-31 16:54:29,809 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e76385ddbd37af190ba2df92b5602e3b/B of e76385ddbd37af190ba2df92b5602e3b into a0883988b1074fa08b9e8bbbfff1b051(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:54:29,810 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e76385ddbd37af190ba2df92b5602e3b: 2023-03-31 16:54:29,810 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b., storeName=e76385ddbd37af190ba2df92b5602e3b/B, priority=13, startTime=1680281669328; duration=0sec 2023-03-31 16:54:29,810 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:54:29,810 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 16:54:29,812 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37172 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 16:54:29,812 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): e76385ddbd37af190ba2df92b5602e3b/C is initiating minor compaction (all files) 2023-03-31 16:54:29,812 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e76385ddbd37af190ba2df92b5602e3b/C in TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:29,813 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/726aa24809b043d6adfb7cd24e6e6817, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/fdbd906ffb1e4b64906a7d7da8de7432, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/5af78e50c5ac4422afdd16d5ba4c5543] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp, totalSize=36.3 K 2023-03-31 16:54:29,813 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 726aa24809b043d6adfb7cd24e6e6817, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=302, earliestPutTs=1680281666864 2023-03-31 16:54:29,815 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting fdbd906ffb1e4b64906a7d7da8de7432, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=320, earliestPutTs=1680281667181 2023-03-31 16:54:29,819 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 5af78e50c5ac4422afdd16d5ba4c5543, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=339, earliestPutTs=1680281668890 2023-03-31 16:54:29,822 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230331daaffc4adbdc49e3b576157b43f4855e to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331daaffc4adbdc49e3b576157b43f4855e 2023-03-31 16:54:29,828 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=356, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/A/66fec5b05aff4e518303dcbe2ff0055e 2023-03-31 16:54:29,831 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): e76385ddbd37af190ba2df92b5602e3b#C#compaction#185 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:54:29,850 DEBUG [Thread-907] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:29,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:29,852 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:29,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:29,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:29,853 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:29,853 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:29,853 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:29,853 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:29,853 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:29,854 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:29,855 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:29,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:29,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:29,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:29,855 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,855 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:29,855 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,855 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:29,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:29,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:29,856 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:29,856 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:29,856 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:29,856 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:29,856 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:29,856 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:29,856 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:29,856 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:29,856 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:29,856 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. started... 2023-03-31 16:54:29,856 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. as already flushing 2023-03-31 16:54:29,856 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:29,856 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:29,856 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:29,857 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,857 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:29,857 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:29,861 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,861 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:29,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:29,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:29,861 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:29,861 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:29,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:29,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:29,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:29,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:29,862 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:29,862 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:29,862 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10b5ab5c[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:29,862 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:29,863 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:29,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:29,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:29,863 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:29,864 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:29,864 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:29,864 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,864 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:29,865 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,865 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:29,865 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:29,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:29,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:29,865 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:29,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:29,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:29,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:29,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:29,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:29,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:29,867 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:29,867 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:29,868 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:29,868 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:29,868 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:29,868 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:29,869 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:29,869 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:29,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:29,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:29,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:29,870 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:29,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:29,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:29,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:29,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:29,870 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:29,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:29,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:29,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:29,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:29,871 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:29,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:29,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,885 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,885 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:29,885 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:29,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:29,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:29,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:29,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:29,885 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:29,885 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:29,885 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:29,885 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:29,885 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,885 DEBUG [Thread-907] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:29,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:29,885 DEBUG [Thread-907] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:29,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:29,885 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:29,886 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:29,886 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:29,986 DEBUG [Thread-907] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:29,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:29,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:29,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:29,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:29,989 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:29,989 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:29,989 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:29,989 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:29,989 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:29,989 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:29,991 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,991 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:29,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:29,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:29,991 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,991 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:29,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:29,991 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:29,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:29,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:29,992 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:29,992 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:29,992 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:29,992 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:29,992 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:29,992 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:29,992 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:29,992 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:29,993 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:29,993 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. started... 2023-03-31 16:54:29,993 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. as already flushing 2023-03-31 16:54:29,993 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:29,993 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:29,993 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:29,993 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,993 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:29,993 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:29,994 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,994 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:29,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:29,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:29,995 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:29,995 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:29,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:29,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:29,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:29,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:29,996 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:29,996 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ed0bbdd[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:29,997 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:29,997 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:29,998 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:29,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:29,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:29,998 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:29,998 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:29,998 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:29,998 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,998 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:29,999 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:29,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52872 deadline: 1680281729997, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:29,999 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:29,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52878 deadline: 1680281729997, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:30,000 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:30,000 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:30,000 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:30,000 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:30,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:30,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:30,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:30,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:30,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:30,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:30,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:30,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:30,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:30,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:30,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:30,004 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:30,004 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:30,004 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:30,004 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:30,004 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:30,004 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:30,006 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:30,006 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:30,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:30,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:30,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:30,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:30,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:30,006 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:30,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:30,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:30,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:30,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:30,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:30,008 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:30,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:30,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:30,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:30,009 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:30,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:30,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:30,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:30,011 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:30,011 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:30,011 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:30,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:30,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:30,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:30,011 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:30,011 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:30,011 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:30,011 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:30,011 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:30,011 DEBUG [Thread-907] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:30,011 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:30,011 DEBUG [Thread-907] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:30,011 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:30,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:30,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:30,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:30,111 DEBUG [Thread-907] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:30,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:30,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:30,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:30,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:30,114 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:30,114 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:30,115 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:30,115 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:30,115 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:30,115 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:30,116 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:30,116 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:30,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:30,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:30,116 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:30,116 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:30,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:30,117 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:30,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:30,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:30,117 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:30,117 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:30,117 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:30,118 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:30,118 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:30,118 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:30,118 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:30,118 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:30,118 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:30,118 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. started... 2023-03-31 16:54:30,118 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. as already flushing 2023-03-31 16:54:30,118 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:30,118 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:30,118 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:30,118 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:30,118 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:30,118 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:30,120 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:30,120 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:30,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:30,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:30,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:30,121 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:30,121 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:30,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:30,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:30,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:30,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:30,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:30,122 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:30,122 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:30,122 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ef3ac0b[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:30,122 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:30,123 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:30,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:30,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:30,124 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:30,124 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:30,124 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:30,124 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:30,124 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:30,125 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:30,125 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:30,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:30,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:30,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:30,125 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:30,125 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:30,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:30,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:30,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:30,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:30,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:30,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:30,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:30,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:30,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:30,128 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:30,128 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:30,128 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:30,128 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:30,128 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:30,129 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:30,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:30,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:30,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:30,130 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:30,130 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:30,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:30,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:30,131 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:30,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:30,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:30,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:30,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:30,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:30,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:30,132 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:30,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:30,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:30,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:30,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:30,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:30,135 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:30,135 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:30,135 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:30,135 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:30,135 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:30,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:30,136 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:30,135 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:30,135 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:30,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:30,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:30,136 DEBUG [Thread-907] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:30,135 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:30,136 DEBUG [Thread-907] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:30,136 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:30,136 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:30,136 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:30,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:30,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:30,230 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/A/9f106f3edb67484eb717c6ebb3a33b5a as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/9f106f3edb67484eb717c6ebb3a33b5a 2023-03-31 16:54:30,234 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e76385ddbd37af190ba2df92b5602e3b/A of e76385ddbd37af190ba2df92b5602e3b into 9f106f3edb67484eb717c6ebb3a33b5a(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:54:30,234 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e76385ddbd37af190ba2df92b5602e3b: 2023-03-31 16:54:30,234 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b., storeName=e76385ddbd37af190ba2df92b5602e3b/A, priority=13, startTime=1680281669328; duration=0sec 2023-03-31 16:54:30,234 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:54:30,236 DEBUG [Thread-907] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:30,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:30,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:30,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:30,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:30,240 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:30,240 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:30,240 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:30,240 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:30,240 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:30,240 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:30,240 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=356 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/B/882169c4d2b24fae8edf094dc9c4fedb 2023-03-31 16:54:30,241 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:30,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:30,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:30,242 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:30,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:30,242 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:30,242 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:30,242 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:30,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:30,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:30,243 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:30,243 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:30,243 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:30,243 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:30,244 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:30,244 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:30,244 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:30,244 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:30,244 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:30,244 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. started... 2023-03-31 16:54:30,244 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. as already flushing 2023-03-31 16:54:30,244 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:30,244 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:30,244 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:30,244 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:30,244 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:30,244 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:30,247 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:30,247 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:30,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:30,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:30,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:30,248 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:30,248 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:30,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:30,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:30,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:30,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:30,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:30,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:30,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e138824[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:30,250 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:30,250 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:30,251 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:30,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:30,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:30,251 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:30,251 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:30,251 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:30,252 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:30,252 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:30,253 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/C/8d7ced1b6a2a4f4ba3d0a46063121ecc as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/8d7ced1b6a2a4f4ba3d0a46063121ecc 2023-03-31 16:54:30,255 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:30,255 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:30,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:30,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:30,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:30,255 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:30,255 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:30,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:30,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:30,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:30,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:30,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:30,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:30,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:30,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:30,259 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:30,259 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:30,259 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:30,259 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:30,259 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:30,259 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:30,261 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=356 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/C/9a94f5d5b3634b90b071c941e11083d2 2023-03-31 16:54:30,261 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:30,261 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:30,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:30,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:30,261 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:30,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:30,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:30,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:30,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:30,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:30,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:30,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:30,262 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:30,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:30,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:30,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:30,263 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:30,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:30,264 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e76385ddbd37af190ba2df92b5602e3b/C of e76385ddbd37af190ba2df92b5602e3b into 8d7ced1b6a2a4f4ba3d0a46063121ecc(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:54:30,265 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e76385ddbd37af190ba2df92b5602e3b: 2023-03-31 16:54:30,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:30,265 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b., storeName=e76385ddbd37af190ba2df92b5602e3b/C, priority=13, startTime=1680281669328; duration=0sec 2023-03-31 16:54:30,265 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:54:30,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:30,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:30,267 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/A/66fec5b05aff4e518303dcbe2ff0055e as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/66fec5b05aff4e518303dcbe2ff0055e 2023-03-31 16:54:30,269 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:30,269 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:30,269 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:30,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:30,270 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:30,269 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:30,270 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:30,270 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:30,270 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:30,270 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:30,270 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:30,270 DEBUG [Thread-907] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:30,269 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:30,270 DEBUG [Thread-907] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:30,270 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:30,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:30,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:30,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:30,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:30,272 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/66fec5b05aff4e518303dcbe2ff0055e, entries=200, sequenceid=356, filesize=32.3 K 2023-03-31 16:54:30,273 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/B/882169c4d2b24fae8edf094dc9c4fedb as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/882169c4d2b24fae8edf094dc9c4fedb 2023-03-31 16:54:30,277 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/882169c4d2b24fae8edf094dc9c4fedb, entries=150, sequenceid=356, filesize=11.9 K 2023-03-31 16:54:30,278 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/C/9a94f5d5b3634b90b071c941e11083d2 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/9a94f5d5b3634b90b071c941e11083d2 2023-03-31 16:54:30,282 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/9a94f5d5b3634b90b071c941e11083d2, entries=150, sequenceid=356, filesize=11.9 K 2023-03-31 16:54:30,283 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for e76385ddbd37af190ba2df92b5602e3b in 929ms, sequenceid=356, compaction requested=false 2023-03-31 16:54:30,283 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e76385ddbd37af190ba2df92b5602e3b: 2023-03-31 16:54:30,370 DEBUG [Thread-907] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:30,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:30,377 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:30,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:30,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:30,378 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:30,378 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:30,378 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:30,378 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:30,378 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:30,378 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:30,379 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:30,379 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:30,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:30,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:30,380 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:30,380 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:30,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:30,380 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:30,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:30,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:30,381 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:30,384 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:30,384 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:30,384 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:30,384 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:30,384 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:30,384 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:30,384 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:30,384 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:30,384 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. started... 2023-03-31 16:54:30,385 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e76385ddbd37af190ba2df92b5602e3b 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-31 16:54:30,385 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e76385ddbd37af190ba2df92b5602e3b, store=A 2023-03-31 16:54:30,385 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:30,385 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e76385ddbd37af190ba2df92b5602e3b, store=B 2023-03-31 16:54:30,385 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:30,385 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e76385ddbd37af190ba2df92b5602e3b, store=C 2023-03-31 16:54:30,385 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:30,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on e76385ddbd37af190ba2df92b5602e3b 2023-03-31 16:54:30,504 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. as already flushing 2023-03-31 16:54:30,553 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:30,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52872 deadline: 1680281730550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:30,553 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:30,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52878 deadline: 1680281730551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:30,655 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:30,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52872 deadline: 1680281730654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:30,655 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:30,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52878 deadline: 1680281730654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:30,812 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023033102ecdb9c58f946a5b0e75f97625ee49d to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033102ecdb9c58f946a5b0e75f97625ee49d 2023-03-31 16:54:30,819 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=374, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/A/29d3b62ab3b5435dab5f2d8487ca86d6 2023-03-31 16:54:30,859 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:30,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52878 deadline: 1680281730857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:30,859 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:30,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52872 deadline: 1680281730858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:31,161 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:31,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52878 deadline: 1680281731160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:31,161 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:31,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52872 deadline: 1680281731161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:31,235 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=374 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/B/0c32e2f85ea340d29b8c54f84d7f5adc 2023-03-31 16:54:31,339 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:31,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52920 deadline: 1680281731339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:31,341 DEBUG [Thread-903] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4139 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b., hostname=jenkins-hbase4.apache.org,33441,1680281619375, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-31 16:54:31,347 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:31,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52892 deadline: 1680281731347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:31,349 DEBUG [Thread-901] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4148 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b., hostname=jenkins-hbase4.apache.org,33441,1680281619375, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-31 16:54:31,646 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=374 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/C/9b06eafeadc54d21b9e48ab8369ddc1e 2023-03-31 16:54:31,661 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/A/29d3b62ab3b5435dab5f2d8487ca86d6 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/29d3b62ab3b5435dab5f2d8487ca86d6 2023-03-31 16:54:31,667 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:31,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52878 deadline: 1680281731664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:31,667 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:31,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52872 deadline: 1680281731667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:31,669 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/29d3b62ab3b5435dab5f2d8487ca86d6, entries=150, sequenceid=374, filesize=25.4 K 2023-03-31 16:54:31,670 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/B/0c32e2f85ea340d29b8c54f84d7f5adc as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/0c32e2f85ea340d29b8c54f84d7f5adc 2023-03-31 16:54:31,675 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/0c32e2f85ea340d29b8c54f84d7f5adc, entries=150, sequenceid=374, filesize=11.9 K 2023-03-31 16:54:31,676 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/C/9b06eafeadc54d21b9e48ab8369ddc1e as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/9b06eafeadc54d21b9e48ab8369ddc1e 2023-03-31 16:54:31,682 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/9b06eafeadc54d21b9e48ab8369ddc1e, entries=150, sequenceid=374, filesize=11.9 K 2023-03-31 16:54:31,683 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for e76385ddbd37af190ba2df92b5602e3b in 1298ms, sequenceid=374, compaction requested=true 2023-03-31 16:54:31,683 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e76385ddbd37af190ba2df92b5602e3b: 2023-03-31 16:54:31,683 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:31,683 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:31,683 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:31,683 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:31,683 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:31,683 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:31,686 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:31,686 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:31,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:31,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:31,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:31,687 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:31,687 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:31,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:31,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:31,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:31,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:31,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:31,690 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:31,690 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@534131f1[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:31,690 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:31,690 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:31,691 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:31,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:31,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:31,691 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:31,691 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:31,692 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:31,692 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:31,692 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:31,693 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:31,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:31,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:31,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:31,693 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:31,693 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:31,693 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:31,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:31,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:31,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:31,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:31,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:31,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:31,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:31,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:31,696 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:31,696 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:31,696 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:31,696 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:31,696 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:31,696 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:31,698 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:31,698 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:31,698 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:31,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:31,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:31,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:31,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:31,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:31,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:31,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:31,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:31,699 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:31,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:31,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:31,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:31,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:31,700 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:31,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:31,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:31,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:31,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:31,704 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:31,704 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:31,704 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:31,704 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:31,704 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:31,704 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:31,704 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:31,704 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:31,704 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:31,705 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:31,705 DEBUG [Thread-907] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:31,705 DEBUG [Thread-907] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:31,705 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:31,705 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:31,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:31,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:31,706 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:31,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:31,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:31,805 DEBUG [Thread-907] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:31,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:31,807 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:31,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:31,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:31,808 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:31,809 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:31,809 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:31,809 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:31,809 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:31,809 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:31,810 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:31,810 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:31,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:31,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:31,811 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:31,811 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:31,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:31,811 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:31,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:31,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:31,811 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:31,812 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:31,812 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:31,812 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:31,812 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:31,812 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:31,812 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:31,812 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:31,812 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:31,812 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. started... 2023-03-31 16:54:31,812 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e76385ddbd37af190ba2df92b5602e3b 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-31 16:54:31,813 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e76385ddbd37af190ba2df92b5602e3b, store=A 2023-03-31 16:54:31,813 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:31,813 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e76385ddbd37af190ba2df92b5602e3b, store=B 2023-03-31 16:54:31,813 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:31,813 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e76385ddbd37af190ba2df92b5602e3b, store=C 2023-03-31 16:54:31,813 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:31,827 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303313322c635dc0f4fd9b108ead450f699c1 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303313322c635dc0f4fd9b108ead450f699c1 2023-03-31 16:54:31,835 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=391, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/A/8df0ca0627984c7cbfc43f21df872db3 2023-03-31 16:54:32,272 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=391 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/B/4709bc6903fd458cba18f1d9dca9dc09 2023-03-31 16:54:32,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on e76385ddbd37af190ba2df92b5602e3b 2023-03-31 16:54:32,674 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. as already flushing 2023-03-31 16:54:32,683 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=391 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/C/35a5b335b7694b9fa35e898b09dc0c03 2023-03-31 16:54:32,688 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/A/8df0ca0627984c7cbfc43f21df872db3 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/8df0ca0627984c7cbfc43f21df872db3 2023-03-31 16:54:32,692 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/8df0ca0627984c7cbfc43f21df872db3, entries=150, sequenceid=391, filesize=25.4 K 2023-03-31 16:54:32,693 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/B/4709bc6903fd458cba18f1d9dca9dc09 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/4709bc6903fd458cba18f1d9dca9dc09 2023-03-31 16:54:32,697 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/4709bc6903fd458cba18f1d9dca9dc09, entries=150, sequenceid=391, filesize=11.9 K 2023-03-31 16:54:32,698 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/C/35a5b335b7694b9fa35e898b09dc0c03 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/35a5b335b7694b9fa35e898b09dc0c03 2023-03-31 16:54:32,702 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:32,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52878 deadline: 1680281732699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:32,702 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/35a5b335b7694b9fa35e898b09dc0c03, entries=150, sequenceid=391, filesize=11.9 K 2023-03-31 16:54:32,702 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:32,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52872 deadline: 1680281732699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:32,703 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=87.22 KB/89310 for e76385ddbd37af190ba2df92b5602e3b in 891ms, sequenceid=391, compaction requested=true 2023-03-31 16:54:32,703 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e76385ddbd37af190ba2df92b5602e3b: 2023-03-31 16:54:32,703 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:32,703 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:32,703 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:32,703 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:32,703 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:32,703 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:32,706 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:32,706 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:32,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:32,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:32,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:32,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:32,707 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:32,707 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:32,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:32,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:32,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:32,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:32,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:32,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38a46617[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:32,708 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:32,708 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:32,711 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:32,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:32,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:32,711 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:32,711 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:32,711 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:32,711 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:32,711 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:32,714 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:32,714 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:32,715 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:32,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:32,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:32,715 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:32,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:32,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:32,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:32,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:32,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:32,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:32,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:32,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:32,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:32,718 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:32,718 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:32,719 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:32,719 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:32,719 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:32,719 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:32,720 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:32,721 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:32,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:32,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:32,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:32,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:32,721 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:32,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:32,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:32,721 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:32,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:32,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:32,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:32,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:32,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:32,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:32,722 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:32,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:32,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:32,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:32,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:32,725 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:32,725 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:32,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:32,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:32,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:32,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:32,726 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:32,726 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:32,726 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:32,726 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:32,726 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:32,726 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:32,726 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:32,726 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:32,726 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:32,726 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:32,726 DEBUG [Thread-907] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:32,727 DEBUG [Thread-907] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:32,727 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:32,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on e76385ddbd37af190ba2df92b5602e3b 2023-03-31 16:54:32,805 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e76385ddbd37af190ba2df92b5602e3b 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-31 16:54:32,805 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e76385ddbd37af190ba2df92b5602e3b, store=A 2023-03-31 16:54:32,806 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:32,806 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e76385ddbd37af190ba2df92b5602e3b, store=B 2023-03-31 16:54:32,806 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:32,806 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e76385ddbd37af190ba2df92b5602e3b, store=C 2023-03-31 16:54:32,806 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:32,827 DEBUG [Thread-907] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:32,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:32,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:32,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:32,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:32,830 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:32,830 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:32,830 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:32,830 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:32,831 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:32,831 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:32,832 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:32,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52872 deadline: 1680281732827, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:32,832 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:32,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52878 deadline: 1680281732830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:32,833 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:32,833 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:32,833 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:32,834 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:32,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:32,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:32,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:32,835 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:32,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:32,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:32,835 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:32,835 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:32,835 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:32,835 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:32,835 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:32,836 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:32,836 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:32,836 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:32,836 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:32,836 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. started... 2023-03-31 16:54:32,836 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. as already flushing 2023-03-31 16:54:32,836 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:32,836 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:32,836 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:32,836 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:32,837 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:32,837 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:32,838 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:32,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:32,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:32,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:32,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:32,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:32,839 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:32,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:32,839 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:32,839 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:32,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:32,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:32,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:32,840 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:32,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7bf416fc[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:32,840 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:32,841 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:32,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:32,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:32,842 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:32,842 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:32,842 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:32,842 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:32,842 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:32,843 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:32,843 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:32,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:32,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:32,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:32,843 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:32,843 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:32,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:32,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:32,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:32,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:32,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:32,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:32,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:32,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:32,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:32,845 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:32,845 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:32,845 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:32,845 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:32,845 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:32,848 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:32,848 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:32,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:32,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:32,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:32,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:32,848 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:32,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:32,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:32,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:32,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:32,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:32,850 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:32,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:32,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:32,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:32,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:32,851 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:32,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:32,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:32,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:32,853 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:32,853 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:32,853 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:32,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:32,853 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:32,853 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:32,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:32,853 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:32,854 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:32,853 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:32,854 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:32,854 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:32,854 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:32,854 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:32,854 DEBUG [Thread-907] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:32,854 DEBUG [Thread-907] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:32,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:32,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:32,913 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:32,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52906 deadline: 1680281732908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:32,914 DEBUG [Thread-905] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18251 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b., hostname=jenkins-hbase4.apache.org,33441,1680281619375, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-31 16:54:32,935 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:32,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52872 deadline: 1680281732933, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:32,936 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:32,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52878 deadline: 1680281732933, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:32,954 DEBUG [Thread-907] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:32,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:32,956 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:32,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:32,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:32,958 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:32,958 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:32,958 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:32,958 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:32,958 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:32,958 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:32,961 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:32,961 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:32,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:32,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:32,962 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:32,962 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:32,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:32,962 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:32,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:32,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:32,963 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:32,963 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:32,963 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:32,963 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:32,963 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:32,963 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:32,963 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:32,963 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:32,963 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:32,963 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. started... 2023-03-31 16:54:32,963 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. as already flushing 2023-03-31 16:54:32,963 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:32,963 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:32,964 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:32,964 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:32,964 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:32,964 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:32,966 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:32,966 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:32,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:32,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:32,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:32,967 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:32,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:32,967 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:32,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:32,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:32,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:32,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:32,968 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:32,968 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9d4e30e[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:32,968 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:32,968 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:32,969 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:32,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:32,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:32,970 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:32,970 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:32,970 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:32,970 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:32,970 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:32,971 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:32,971 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:32,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:32,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:32,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:32,971 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:32,971 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:32,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:32,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:32,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:32,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:32,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:32,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:32,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:32,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:32,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:32,973 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:32,973 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:32,974 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:32,974 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:32,974 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:32,975 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:32,975 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:32,975 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:32,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:32,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:32,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:32,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:32,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:32,975 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:32,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:32,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:32,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:32,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:32,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:32,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:32,977 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:32,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:32,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:32,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:32,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:32,979 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:32,979 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:32,979 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:32,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:32,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:32,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:32,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:32,979 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:32,979 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:32,980 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:32,979 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:32,980 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:32,980 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:32,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:32,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:32,980 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:32,980 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:32,981 DEBUG [Thread-907] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:32,981 DEBUG [Thread-907] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:33,081 DEBUG [Thread-907] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:33,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:33,083 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:33,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:33,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:33,084 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:33,084 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:33,084 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:33,084 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:33,085 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:33,085 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:33,086 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:33,086 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:33,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:33,086 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,086 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:33,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:33,087 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:33,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:33,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:33,087 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:33,087 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:33,088 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:33,088 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:33,088 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:33,088 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:33,088 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:33,088 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:33,088 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:33,088 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. started... 2023-03-31 16:54:33,088 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. as already flushing 2023-03-31 16:54:33,088 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:33,088 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:33,088 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:33,088 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,088 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:33,088 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:33,090 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,090 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:33,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:33,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:33,090 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:33,090 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:33,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:33,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:33,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:33,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:33,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:33,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5340957c[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:33,092 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:33,092 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:33,093 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:33,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:33,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:33,093 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:33,093 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:33,093 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:33,093 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,093 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:33,095 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:33,095 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:33,095 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:33,095 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:33,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:33,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:33,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:33,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:33,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:33,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:33,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:33,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:33,098 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:33,098 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:33,098 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:33,098 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:33,098 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:33,099 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:33,099 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:33,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:33,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:33,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:33,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:33,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:33,100 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:33,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:33,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:33,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:33,100 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:33,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:33,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:33,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:33,101 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:33,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:33,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,104 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,104 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:33,104 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:33,104 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,104 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:33,104 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:33,104 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:33,104 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:33,104 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:33,104 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:33,104 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:33,105 DEBUG [Thread-907] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:33,105 DEBUG [Thread-907] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:33,105 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:33,105 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:33,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:33,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:33,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:33,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:33,137 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:33,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52872 deadline: 1680281733136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,140 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:33,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52878 deadline: 1680281733138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,205 DEBUG [Thread-907] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:33,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:33,207 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:33,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:33,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:33,208 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:33,208 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:33,208 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:33,208 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:33,208 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:33,209 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:33,210 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:33,210 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:33,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:33,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:33,210 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,210 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:33,210 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:33,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:33,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:33,211 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:33,211 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:33,211 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:33,211 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:33,211 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:33,211 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:33,211 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:33,212 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:33,212 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:33,212 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. started... 2023-03-31 16:54:33,212 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. as already flushing 2023-03-31 16:54:33,212 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:33,212 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:33,212 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:33,212 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,212 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:33,212 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:33,214 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:33,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:33,215 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:33,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:33,215 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:33,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:33,215 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:33,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:33,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:33,216 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:33,217 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@484fd5dd[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:33,217 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:33,217 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:33,218 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:33,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:33,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:33,218 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:33,218 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:33,218 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:33,218 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,219 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:33,219 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:33,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:33,219 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:33,220 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:33,220 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023033103ba35a5fd3b41fea5fe7b8c1b53d74b to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033103ba35a5fd3b41fea5fe7b8c1b53d74b 2023-03-31 16:54:33,220 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:33,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:33,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:33,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:33,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:33,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:33,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:33,222 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:33,222 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:33,222 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:33,222 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:33,222 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:33,222 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:33,223 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:33,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:33,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:33,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:33,224 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:33,224 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:33,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:33,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:33,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:33,224 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:33,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:33,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:33,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:33,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:33,226 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:33,229 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,229 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:33,229 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:33,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:33,229 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:33,229 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,229 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:33,229 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:33,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:33,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:33,229 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:33,229 DEBUG [Thread-907] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:33,229 DEBUG [Thread-907] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:33,229 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:33,229 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:33,230 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:33,230 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:33,230 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:33,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:33,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:33,329 DEBUG [Thread-907] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:33,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:33,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:33,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:33,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:33,332 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:33,333 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:33,333 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:33,333 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:33,333 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:33,333 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:33,334 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,334 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,334 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:33,336 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:33,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:33,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:33,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:33,336 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:33,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:33,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:33,337 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:33,337 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:33,337 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:33,337 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:33,337 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:33,337 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:33,337 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:33,337 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:33,337 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:33,338 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. started... 2023-03-31 16:54:33,338 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. as already flushing 2023-03-31 16:54:33,338 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:33,338 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:33,338 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:33,338 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,338 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:33,338 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:33,339 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,340 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:33,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:33,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:33,340 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:33,340 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:33,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:33,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:33,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:33,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:33,341 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:33,341 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13d779de[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:33,341 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:33,341 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:33,346 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:33,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:33,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:33,346 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:33,346 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:33,346 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:33,346 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,346 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:33,347 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:33,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:33,347 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:33,347 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:33,348 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:33,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:33,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:33,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:33,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:33,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:33,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:33,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:33,350 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:33,350 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:33,350 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:33,350 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:33,350 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:33,351 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:33,351 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:33,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:33,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:33,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:33,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:33,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:33,351 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:33,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:33,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:33,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:33,352 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:33,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:33,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:33,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:33,354 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:33,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:33,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,356 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,356 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:33,356 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:33,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:33,356 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:33,356 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:33,356 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:33,356 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:33,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:33,356 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:33,356 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,356 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:33,357 DEBUG [Thread-907] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:33,357 DEBUG [Thread-907] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:33,357 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:33,357 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:33,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:33,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:33,357 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:33,439 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:33,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52872 deadline: 1680281733439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,445 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:33,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52878 deadline: 1680281733443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,457 DEBUG [Thread-907] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:33,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:33,459 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:33,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:33,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:33,460 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:33,460 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:33,460 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:33,460 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:33,460 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:33,460 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:33,462 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,462 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:33,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:33,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:33,463 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,463 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:33,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:33,463 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:33,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:33,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:33,463 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:33,464 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:33,464 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:33,464 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:33,464 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:33,464 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:33,464 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:33,464 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:33,464 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:33,464 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. started... 2023-03-31 16:54:33,464 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. as already flushing 2023-03-31 16:54:33,464 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:33,464 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:33,464 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:33,464 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,464 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:33,464 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:33,466 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,466 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:33,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:33,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:33,466 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:33,466 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:33,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:33,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:33,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:33,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:33,467 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:33,467 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:33,467 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3db52439[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:33,467 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:33,468 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:33,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:33,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:33,469 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:33,469 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:33,469 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:33,469 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,469 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:33,470 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,470 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:33,470 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:33,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:33,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:33,470 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:33,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:33,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:33,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:33,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:33,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:33,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:33,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:33,473 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:33,473 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:33,473 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:33,473 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:33,473 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:33,476 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:33,476 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:33,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:33,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:33,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:33,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:33,476 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:33,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:33,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:33,477 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:33,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:33,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:33,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:33,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:33,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:33,478 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:33,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:33,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,481 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,481 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:33,481 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:33,481 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:33,481 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:33,481 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:33,481 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:33,481 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:33,481 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,481 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:33,481 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:33,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:33,481 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:33,481 DEBUG [Thread-907] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:33,481 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:33,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:33,481 DEBUG [Thread-907] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:33,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:33,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:33,582 DEBUG [Thread-907] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:33,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:33,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:33,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:33,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:33,585 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:33,585 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:33,585 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:33,585 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:33,585 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:33,585 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:33,587 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:33,587 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:33,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:33,587 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,587 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:33,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:33,587 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:33,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:33,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:33,588 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:33,588 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:33,588 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:33,588 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:33,588 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:33,588 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:33,588 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:33,588 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:33,588 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:33,588 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. started... 2023-03-31 16:54:33,589 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. as already flushing 2023-03-31 16:54:33,589 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:33,589 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:33,589 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:33,589 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,589 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:33,589 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:33,590 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:33,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:33,591 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:33,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:33,591 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:33,591 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:33,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:33,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:33,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:33,592 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:33,592 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:33,592 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a09ad86[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:33,592 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:33,594 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:33,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:33,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:33,595 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:33,595 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:33,595 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:33,595 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,595 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:33,596 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,596 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:33,596 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:33,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:33,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:33,596 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:33,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:33,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:33,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:33,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:33,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:33,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:33,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:33,598 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:33,598 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:33,598 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:33,598 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:33,598 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:33,600 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:33,600 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:33,600 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:33,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:33,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:33,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:33,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:33,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:33,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:33,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:33,601 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:33,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:33,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:33,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:33,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:33,602 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:33,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:33,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,605 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,605 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:33,605 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:33,605 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:33,605 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:33,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:33,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:33,605 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,605 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:33,605 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:33,605 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:33,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:33,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:33,605 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:33,605 DEBUG [Thread-907] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:33,606 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:33,606 DEBUG [Thread-907] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:33,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:33,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:33,626 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=409, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/A/dc6043eaae2b4c26abc32cd24743a78b 2023-03-31 16:54:33,706 DEBUG [Thread-907] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:33,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:33,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:33,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:33,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:33,709 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:33,709 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:33,709 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:33,709 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:33,710 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:33,710 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:33,711 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,711 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:33,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:33,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:33,711 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,711 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:33,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:33,712 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:33,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:33,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:33,712 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:33,712 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:33,712 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:33,712 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:33,712 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:33,712 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:33,713 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:33,713 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:33,713 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:33,713 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. started... 2023-03-31 16:54:33,713 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. as already flushing 2023-03-31 16:54:33,713 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:33,713 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:33,713 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:33,713 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,713 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:33,713 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:33,714 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,714 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:33,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:33,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:33,715 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:33,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:33,715 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:33,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:33,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:33,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:33,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:33,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c5e6c27[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:33,716 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:33,716 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:33,717 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:33,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:33,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:33,717 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:33,717 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:33,717 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:33,717 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,717 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:33,722 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:33,722 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,722 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:33,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:33,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:33,722 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:33,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:33,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:33,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:33,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:33,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:33,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:33,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:33,724 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:33,725 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:33,725 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:33,725 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:33,725 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:33,726 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:33,726 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:33,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:33,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:33,726 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:33,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:33,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:33,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:33,727 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:33,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:33,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:33,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:33,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:33,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:33,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:33,728 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:33,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:33,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,730 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:33,730 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:33,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:33,730 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:33,730 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:33,730 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:33,730 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:33,730 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:33,730 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:33,730 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:33,730 DEBUG [Thread-907] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:33,730 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:33,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:33,731 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:33,731 DEBUG [Thread-907] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:33,731 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:33,731 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:33,831 DEBUG [Thread-907] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:33,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:33,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:33,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:33,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:33,834 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:33,834 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:33,834 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:33,834 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:33,834 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:33,834 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:33,836 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:33,836 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:33,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:33,837 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,837 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:33,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:33,837 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:33,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:33,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:33,837 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:33,837 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:33,838 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:33,838 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:33,838 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:33,838 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:33,838 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:33,838 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:33,838 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:33,838 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. started... 2023-03-31 16:54:33,838 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. as already flushing 2023-03-31 16:54:33,838 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:33,838 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:33,838 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:33,838 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,838 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:33,838 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:33,840 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:33,840 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:33,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:33,840 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:33,840 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:33,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:33,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:33,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:33,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:33,841 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:33,841 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74eff7fb[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:33,841 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:33,841 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:33,842 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:33,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:33,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:33,843 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:33,843 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:33,843 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:33,843 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,843 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:33,844 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,844 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:33,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:33,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:33,844 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:33,844 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:33,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:33,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:33,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:33,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:33,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:33,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:33,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:33,846 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:33,847 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:33,847 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:33,847 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:33,847 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:33,849 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:33,850 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:33,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:33,850 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:33,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:33,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:33,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:33,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:33,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:33,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:33,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:33,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:33,851 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:33,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:33,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:33,852 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:33,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:33,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,854 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:33,854 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:33,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:33,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:33,854 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:33,854 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,854 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:33,854 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:33,854 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:33,854 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:33,854 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,854 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:33,854 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:33,854 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:33,854 DEBUG [Thread-907] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:33,854 DEBUG [Thread-907] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:33,854 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:33,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:33,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:33,943 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:33,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52872 deadline: 1680281733943, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,945 DEBUG [Thread-910] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x508246cc to 127.0.0.1:51916 2023-03-31 16:54:33,945 DEBUG [Thread-910] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 16:54:33,946 DEBUG [Thread-914] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x58e27a32 to 127.0.0.1:51916 2023-03-31 16:54:33,947 DEBUG [Thread-914] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 16:54:33,948 DEBUG [Thread-908] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x00c2598b to 127.0.0.1:51916 2023-03-31 16:54:33,948 DEBUG [Thread-908] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 16:54:33,950 DEBUG [Thread-912] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x08795451 to 127.0.0.1:51916 2023-03-31 16:54:33,950 DEBUG [Thread-912] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 16:54:33,951 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:33,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52878 deadline: 1680281733950, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:33,954 DEBUG [Thread-907] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:33,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:34,036 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=409 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/B/87647f51c71a43edb71d22fc6d788c48 2023-03-31 16:54:34,446 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=409 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/C/9202765617f04e0da9cf06b5289d30c6 2023-03-31 16:54:34,450 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/A/dc6043eaae2b4c26abc32cd24743a78b as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/dc6043eaae2b4c26abc32cd24743a78b 2023-03-31 16:54:34,453 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/dc6043eaae2b4c26abc32cd24743a78b, entries=200, sequenceid=409, filesize=32.3 K 2023-03-31 16:54:34,454 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/B/87647f51c71a43edb71d22fc6d788c48 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/87647f51c71a43edb71d22fc6d788c48 2023-03-31 16:54:34,457 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/87647f51c71a43edb71d22fc6d788c48, entries=150, sequenceid=409, filesize=11.9 K 2023-03-31 16:54:34,458 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/C/9202765617f04e0da9cf06b5289d30c6 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/9202765617f04e0da9cf06b5289d30c6 2023-03-31 16:54:34,461 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/9202765617f04e0da9cf06b5289d30c6, entries=150, sequenceid=409, filesize=11.9 K 2023-03-31 16:54:34,462 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for e76385ddbd37af190ba2df92b5602e3b in 1657ms, sequenceid=409, compaction requested=true 2023-03-31 16:54:34,462 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e76385ddbd37af190ba2df92b5602e3b: 2023-03-31 16:54:34,462 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:54:34,462 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-31 16:54:34,462 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-31 16:54:34,462 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:54:34,462 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:54:34,463 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 145026 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-31 16:54:34,464 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1912): e76385ddbd37af190ba2df92b5602e3b/A is initiating minor compaction (all files) 2023-03-31 16:54:34,464 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61646 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-31 16:54:34,464 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e76385ddbd37af190ba2df92b5602e3b/A in TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:34,464 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): e76385ddbd37af190ba2df92b5602e3b/B is initiating minor compaction (all files) 2023-03-31 16:54:34,464 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/9f106f3edb67484eb717c6ebb3a33b5a, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/66fec5b05aff4e518303dcbe2ff0055e, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/29d3b62ab3b5435dab5f2d8487ca86d6, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/8df0ca0627984c7cbfc43f21df872db3, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/dc6043eaae2b4c26abc32cd24743a78b] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp, totalSize=141.6 K 2023-03-31 16:54:34,464 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e76385ddbd37af190ba2df92b5602e3b/B in TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:34,464 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/a0883988b1074fa08b9e8bbbfff1b051, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/882169c4d2b24fae8edf094dc9c4fedb, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/0c32e2f85ea340d29b8c54f84d7f5adc, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/4709bc6903fd458cba18f1d9dca9dc09, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/87647f51c71a43edb71d22fc6d788c48] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp, totalSize=60.2 K 2023-03-31 16:54:34,464 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 9f106f3edb67484eb717c6ebb3a33b5a, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=339, earliestPutTs=1680281668890 2023-03-31 16:54:34,464 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting a0883988b1074fa08b9e8bbbfff1b051, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=339, earliestPutTs=1680281668890 2023-03-31 16:54:34,465 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 66fec5b05aff4e518303dcbe2ff0055e, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=356, earliestPutTs=1680281669243 2023-03-31 16:54:34,465 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 882169c4d2b24fae8edf094dc9c4fedb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=356, earliestPutTs=1680281669243 2023-03-31 16:54:34,465 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 29d3b62ab3b5435dab5f2d8487ca86d6, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=374, earliestPutTs=1680281669376 2023-03-31 16:54:34,465 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 0c32e2f85ea340d29b8c54f84d7f5adc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=374, earliestPutTs=1680281669376 2023-03-31 16:54:34,465 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 8df0ca0627984c7cbfc43f21df872db3, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=391, earliestPutTs=1680281670506 2023-03-31 16:54:34,465 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 4709bc6903fd458cba18f1d9dca9dc09, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=391, earliestPutTs=1680281670506 2023-03-31 16:54:34,466 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting dc6043eaae2b4c26abc32cd24743a78b, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=409, earliestPutTs=1680281672691 2023-03-31 16:54:34,466 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 87647f51c71a43edb71d22fc6d788c48, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=409, earliestPutTs=1680281672697 2023-03-31 16:54:34,474 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): e76385ddbd37af190ba2df92b5602e3b#B#compaction#198 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:54:34,481 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] throttle.PressureAwareThroughputController(145): e76385ddbd37af190ba2df92b5602e3b#A#compaction#197 average throughput is 2.46 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:54:34,887 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/B/cf988a9a42244d699dba69494ae0185c as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/cf988a9a42244d699dba69494ae0185c 2023-03-31 16:54:34,892 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in e76385ddbd37af190ba2df92b5602e3b/B of e76385ddbd37af190ba2df92b5602e3b into cf988a9a42244d699dba69494ae0185c(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:54:34,892 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e76385ddbd37af190ba2df92b5602e3b: 2023-03-31 16:54:34,892 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b., storeName=e76385ddbd37af190ba2df92b5602e3b/B, priority=11, startTime=1680281674462; duration=0sec 2023-03-31 16:54:34,892 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:54:34,892 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-31 16:54:34,894 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61646 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-31 16:54:34,894 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): e76385ddbd37af190ba2df92b5602e3b/C is initiating minor compaction (all files) 2023-03-31 16:54:34,894 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e76385ddbd37af190ba2df92b5602e3b/C in TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:34,894 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/8d7ced1b6a2a4f4ba3d0a46063121ecc, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/9a94f5d5b3634b90b071c941e11083d2, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/9b06eafeadc54d21b9e48ab8369ddc1e, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/35a5b335b7694b9fa35e898b09dc0c03, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/9202765617f04e0da9cf06b5289d30c6] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp, totalSize=60.2 K 2023-03-31 16:54:34,895 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 8d7ced1b6a2a4f4ba3d0a46063121ecc, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=339, earliestPutTs=1680281668890 2023-03-31 16:54:34,895 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 9a94f5d5b3634b90b071c941e11083d2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=356, earliestPutTs=1680281669243 2023-03-31 16:54:34,895 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 9b06eafeadc54d21b9e48ab8369ddc1e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=374, earliestPutTs=1680281669376 2023-03-31 16:54:34,896 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 35a5b335b7694b9fa35e898b09dc0c03, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=391, earliestPutTs=1680281670506 2023-03-31 16:54:34,896 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 9202765617f04e0da9cf06b5289d30c6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=409, earliestPutTs=1680281672697 2023-03-31 16:54:34,897 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/A/757788f2184849f99ffa5d0ccf9771b3 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/757788f2184849f99ffa5d0ccf9771b3 2023-03-31 16:54:34,901 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in e76385ddbd37af190ba2df92b5602e3b/A of e76385ddbd37af190ba2df92b5602e3b into 757788f2184849f99ffa5d0ccf9771b3(size=26.3 K), total size for store is 26.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:54:34,901 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e76385ddbd37af190ba2df92b5602e3b: 2023-03-31 16:54:34,901 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b., storeName=e76385ddbd37af190ba2df92b5602e3b/A, priority=11, startTime=1680281674462; duration=0sec 2023-03-31 16:54:34,902 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:54:34,904 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): e76385ddbd37af190ba2df92b5602e3b#C#compaction#199 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:54:34,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on e76385ddbd37af190ba2df92b5602e3b 2023-03-31 16:54:34,952 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e76385ddbd37af190ba2df92b5602e3b 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-31 16:54:34,952 DEBUG [Thread-899] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0a71edf0 to 127.0.0.1:51916 2023-03-31 16:54:34,952 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e76385ddbd37af190ba2df92b5602e3b, store=A 2023-03-31 16:54:34,952 DEBUG [Thread-899] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 16:54:34,952 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:34,952 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e76385ddbd37af190ba2df92b5602e3b, store=B 2023-03-31 16:54:34,952 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:34,952 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e76385ddbd37af190ba2df92b5602e3b, store=C 2023-03-31 16:54:34,952 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:34,957 DEBUG [Thread-897] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x051c62b6 to 127.0.0.1:51916 2023-03-31 16:54:34,957 DEBUG [Thread-897] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 16:54:35,312 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/C/ec3565a623d049619c1d9a48d20495b9 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/ec3565a623d049619c1d9a48d20495b9 2023-03-31 16:54:35,316 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in e76385ddbd37af190ba2df92b5602e3b/C of e76385ddbd37af190ba2df92b5602e3b into ec3565a623d049619c1d9a48d20495b9(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:54:35,316 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e76385ddbd37af190ba2df92b5602e3b: 2023-03-31 16:54:35,316 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b., storeName=e76385ddbd37af190ba2df92b5602e3b/C, priority=11, startTime=1680281674462; duration=0sec 2023-03-31 16:54:35,316 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:54:35,354 DEBUG [Thread-901] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x21265fcb to 127.0.0.1:51916 2023-03-31 16:54:35,354 DEBUG [Thread-901] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 16:54:35,363 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230331907d95d7dd85492b983f6fb2016e63b4 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331907d95d7dd85492b983f6fb2016e63b4 2023-03-31 16:54:35,378 DEBUG [Thread-903] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7aba8e46 to 127.0.0.1:51916 2023-03-31 16:54:35,378 DEBUG [Thread-903] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 16:54:35,768 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=426, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/A/6b41ce19e46f4daca32517cbf2127b19 2023-03-31 16:54:36,179 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=426 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/B/d652020d6643414fa94fa644a45449bb 2023-03-31 16:54:36,588 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=426 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/C/0fcbe793ed44473189dced508b71ef41 2023-03-31 16:54:36,593 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/A/6b41ce19e46f4daca32517cbf2127b19 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/6b41ce19e46f4daca32517cbf2127b19 2023-03-31 16:54:36,597 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/6b41ce19e46f4daca32517cbf2127b19, entries=150, sequenceid=426, filesize=25.4 K 2023-03-31 16:54:36,598 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/B/d652020d6643414fa94fa644a45449bb as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/d652020d6643414fa94fa644a45449bb 2023-03-31 16:54:36,602 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/d652020d6643414fa94fa644a45449bb, entries=150, sequenceid=426, filesize=11.9 K 2023-03-31 16:54:36,602 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/C/0fcbe793ed44473189dced508b71ef41 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/0fcbe793ed44473189dced508b71ef41 2023-03-31 16:54:36,607 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/0fcbe793ed44473189dced508b71ef41, entries=150, sequenceid=426, filesize=11.9 K 2023-03-31 16:54:36,607 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=20.13 KB/20610 for e76385ddbd37af190ba2df92b5602e3b in 1655ms, sequenceid=426, compaction requested=false 2023-03-31 16:54:36,608 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e76385ddbd37af190ba2df92b5602e3b: 2023-03-31 16:54:40,686 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-03-31 16:54:42,954 DEBUG [Thread-905] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0feb770d to 127.0.0.1:51916 2023-03-31 16:54:42,954 DEBUG [Thread-905] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 16:54:42,954 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-31 16:54:42,954 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(394): wrote 94 2023-03-31 16:54:42,954 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(394): wrote 96 2023-03-31 16:54:42,954 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(394): wrote 65 2023-03-31 16:54:42,954 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(394): wrote 69 2023-03-31 16:54:42,954 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(394): wrote 15 2023-03-31 16:54:42,954 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-31 16:54:42,954 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(398): read 5095 2023-03-31 16:54:42,954 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(398): read 5112 2023-03-31 16:54:42,954 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-31 16:54:42,954 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(402): scanned 2131 2023-03-31 16:54:42,954 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(403): verified 6393 rows 2023-03-31 16:54:42,954 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(402): scanned 2121 2023-03-31 16:54:42,955 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(403): verified 6359 rows 2023-03-31 16:54:42,955 INFO [Listener at localhost/44135] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-31 16:54:42,955 DEBUG [Listener at localhost/44135] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5b8d0eb2 to 127.0.0.1:51916 2023-03-31 16:54:42,955 DEBUG [Listener at localhost/44135] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 16:54:42,956 INFO [Listener at localhost/44135] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-31 16:54:42,957 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-03-31 16:54:42,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure2.ProcedureExecutor(1029): Stored pid=24, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-31 16:54:42,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-31 16:54:42,961 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680281682961"}]},"ts":"1680281682961"} 2023-03-31 16:54:42,962 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-31 16:54:42,965 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-31 16:54:42,965 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e76385ddbd37af190ba2df92b5602e3b, UNASSIGN}] 2023-03-31 16:54:42,966 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e76385ddbd37af190ba2df92b5602e3b, UNASSIGN 2023-03-31 16:54:42,967 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=e76385ddbd37af190ba2df92b5602e3b, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:42,967 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680281682967"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680281682967"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680281682967"}]},"ts":"1680281682967"} 2023-03-31 16:54:42,968 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=25, state=RUNNABLE; CloseRegionProcedure e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375}] 2023-03-31 16:54:43,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-31 16:54:43,120 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close e76385ddbd37af190ba2df92b5602e3b 2023-03-31 16:54:43,120 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing e76385ddbd37af190ba2df92b5602e3b, disabling compactions & flushes 2023-03-31 16:54:43,120 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:43,120 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:43,120 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. after waiting 0 ms 2023-03-31 16:54:43,120 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:43,120 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing e76385ddbd37af190ba2df92b5602e3b 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-03-31 16:54:43,120 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e76385ddbd37af190ba2df92b5602e3b, store=A 2023-03-31 16:54:43,120 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:43,120 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e76385ddbd37af190ba2df92b5602e3b, store=B 2023-03-31 16:54:43,120 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:43,120 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e76385ddbd37af190ba2df92b5602e3b, store=C 2023-03-31 16:54:43,120 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:43,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-31 16:54:43,533 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230331e67ff55bf1fb4b2da18958d4d8f1df08 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331e67ff55bf1fb4b2da18958d4d8f1df08 2023-03-31 16:54:43,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-31 16:54:43,938 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=434, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/A/cc699c0c12774ebb9b669de5d317e0b8 2023-03-31 16:54:43,948 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=434 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/B/fbbfa65fb92247fc8ea30a1b2313df3d 2023-03-31 16:54:44,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-31 16:54:44,358 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=434 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/C/8778daef02cc42c6b8c18820b4c185a9 2023-03-31 16:54:44,362 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/A/cc699c0c12774ebb9b669de5d317e0b8 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/cc699c0c12774ebb9b669de5d317e0b8 2023-03-31 16:54:44,365 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/cc699c0c12774ebb9b669de5d317e0b8, entries=100, sequenceid=434, filesize=18.6 K 2023-03-31 16:54:44,366 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/B/fbbfa65fb92247fc8ea30a1b2313df3d as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/fbbfa65fb92247fc8ea30a1b2313df3d 2023-03-31 16:54:44,370 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/fbbfa65fb92247fc8ea30a1b2313df3d, entries=100, sequenceid=434, filesize=9.5 K 2023-03-31 16:54:44,370 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/.tmp/C/8778daef02cc42c6b8c18820b4c185a9 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/8778daef02cc42c6b8c18820b4c185a9 2023-03-31 16:54:44,374 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/8778daef02cc42c6b8c18820b4c185a9, entries=100, sequenceid=434, filesize=9.5 K 2023-03-31 16:54:44,374 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for e76385ddbd37af190ba2df92b5602e3b in 1254ms, sequenceid=434, compaction requested=true 2023-03-31 16:54:44,395 DEBUG [StoreCloser-TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/22abf05162964cb698f718f8d8a09162, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/caf1fcac7bcf41e99bede9c8d7470dfb, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/7125aeff4e0e423282a4bf05dc0fd495, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/30aa652862a947dfbc88722a0caa6743, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/9ae7627fbf8b4536a2a289cfae4492ed, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/a1beb4ab665c4d69a714f89128eaa846, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/9123603f0f8b4a14a2be8620eeda5855, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/d76d45ffd225438482ce156f4161d195, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/0a78cf1961d043a59eba0b61848a3f43, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/c0e7d3daf0b04c6cab9c84a9c888b3cd, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/c8527ce4bbff417daff8269fbdd037d8, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/62c7371e752d4e0388488a49fc7c2844, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/c34ee73e86f74737846647f622780292, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/08825b5aafa943f19831152a4a83b070, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/a14f29985ffd48d4b519bb35864b2493, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/1b92ac5c3d7b4befad2f12095f9ef1e1, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/47141897df044eca95b05aed894c5d18, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/2500ad05fb71487bb308e2b2a9f60c4c, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/536a348080da40f4b72f1ad98f22c240, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/4f4a1afb2eb140d6a452686e00497ecd, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/ffea3aea26b5493a93ddda4a6f984be3, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/9f106f3edb67484eb717c6ebb3a33b5a, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/66fec5b05aff4e518303dcbe2ff0055e, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/29d3b62ab3b5435dab5f2d8487ca86d6, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/8df0ca0627984c7cbfc43f21df872db3, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/dc6043eaae2b4c26abc32cd24743a78b] to archive 2023-03-31 16:54:44,397 DEBUG [StoreCloser-TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-31 16:54:44,399 DEBUG [StoreCloser-TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/22abf05162964cb698f718f8d8a09162 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/22abf05162964cb698f718f8d8a09162 2023-03-31 16:54:44,401 DEBUG [StoreCloser-TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/caf1fcac7bcf41e99bede9c8d7470dfb to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/caf1fcac7bcf41e99bede9c8d7470dfb 2023-03-31 16:54:44,402 DEBUG [StoreCloser-TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/7125aeff4e0e423282a4bf05dc0fd495 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/7125aeff4e0e423282a4bf05dc0fd495 2023-03-31 16:54:44,403 DEBUG [StoreCloser-TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/30aa652862a947dfbc88722a0caa6743 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/30aa652862a947dfbc88722a0caa6743 2023-03-31 16:54:44,405 DEBUG [StoreCloser-TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/9ae7627fbf8b4536a2a289cfae4492ed to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/9ae7627fbf8b4536a2a289cfae4492ed 2023-03-31 16:54:44,406 DEBUG [StoreCloser-TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/a1beb4ab665c4d69a714f89128eaa846 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/a1beb4ab665c4d69a714f89128eaa846 2023-03-31 16:54:44,407 DEBUG [StoreCloser-TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/9123603f0f8b4a14a2be8620eeda5855 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/9123603f0f8b4a14a2be8620eeda5855 2023-03-31 16:54:44,408 DEBUG [StoreCloser-TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/d76d45ffd225438482ce156f4161d195 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/d76d45ffd225438482ce156f4161d195 2023-03-31 16:54:44,409 DEBUG [StoreCloser-TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/0a78cf1961d043a59eba0b61848a3f43 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/0a78cf1961d043a59eba0b61848a3f43 2023-03-31 16:54:44,410 DEBUG [StoreCloser-TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/c0e7d3daf0b04c6cab9c84a9c888b3cd to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/c0e7d3daf0b04c6cab9c84a9c888b3cd 2023-03-31 16:54:44,411 DEBUG [StoreCloser-TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/c8527ce4bbff417daff8269fbdd037d8 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/c8527ce4bbff417daff8269fbdd037d8 2023-03-31 16:54:44,412 DEBUG [StoreCloser-TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/62c7371e752d4e0388488a49fc7c2844 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/62c7371e752d4e0388488a49fc7c2844 2023-03-31 16:54:44,413 DEBUG [StoreCloser-TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/c34ee73e86f74737846647f622780292 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/c34ee73e86f74737846647f622780292 2023-03-31 16:54:44,415 DEBUG [StoreCloser-TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/08825b5aafa943f19831152a4a83b070 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/08825b5aafa943f19831152a4a83b070 2023-03-31 16:54:44,416 DEBUG [StoreCloser-TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/a14f29985ffd48d4b519bb35864b2493 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/a14f29985ffd48d4b519bb35864b2493 2023-03-31 16:54:44,417 DEBUG [StoreCloser-TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/1b92ac5c3d7b4befad2f12095f9ef1e1 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/1b92ac5c3d7b4befad2f12095f9ef1e1 2023-03-31 16:54:44,418 DEBUG [StoreCloser-TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/47141897df044eca95b05aed894c5d18 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/47141897df044eca95b05aed894c5d18 2023-03-31 16:54:44,419 DEBUG [StoreCloser-TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/2500ad05fb71487bb308e2b2a9f60c4c to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/2500ad05fb71487bb308e2b2a9f60c4c 2023-03-31 16:54:44,420 DEBUG [StoreCloser-TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/536a348080da40f4b72f1ad98f22c240 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/536a348080da40f4b72f1ad98f22c240 2023-03-31 16:54:44,421 DEBUG [StoreCloser-TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/4f4a1afb2eb140d6a452686e00497ecd to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/4f4a1afb2eb140d6a452686e00497ecd 2023-03-31 16:54:44,422 DEBUG [StoreCloser-TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/ffea3aea26b5493a93ddda4a6f984be3 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/ffea3aea26b5493a93ddda4a6f984be3 2023-03-31 16:54:44,423 DEBUG [StoreCloser-TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/9f106f3edb67484eb717c6ebb3a33b5a to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/9f106f3edb67484eb717c6ebb3a33b5a 2023-03-31 16:54:44,424 DEBUG [StoreCloser-TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/66fec5b05aff4e518303dcbe2ff0055e to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/66fec5b05aff4e518303dcbe2ff0055e 2023-03-31 16:54:44,425 DEBUG [StoreCloser-TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/29d3b62ab3b5435dab5f2d8487ca86d6 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/29d3b62ab3b5435dab5f2d8487ca86d6 2023-03-31 16:54:44,426 DEBUG [StoreCloser-TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/8df0ca0627984c7cbfc43f21df872db3 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/8df0ca0627984c7cbfc43f21df872db3 2023-03-31 16:54:44,427 DEBUG [StoreCloser-TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/dc6043eaae2b4c26abc32cd24743a78b to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/dc6043eaae2b4c26abc32cd24743a78b 2023-03-31 16:54:44,430 DEBUG [StoreCloser-TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/bb3f5d3f50584c91a07b0498a7d638eb, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/9618f2346f0e4219afe0af8176a26ca9, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/8ec4b187750e43ada713cc6e720ee32c, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/cffe11007d554e74ba6f7b376f31e142, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/b87470db93dc46e7b06273abafeb16b9, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/074f475d70214880a9bbec74f8fc49b6, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/0fc1625fa8f74442bb50483deca7510c, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/b1c0303b69d445a3ad1ebd4679ebbd84, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/8239929e2581489190769f99bb68cbb3, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/7aec5dbcc7034377965cdb4265fe079d, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/b21f40184b63425283a74bb12189bf64, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/e0e35469f2904e61bc0de0b657e04362, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/08fdc2d8e0ac4355bd611f9c1370ece5, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/1f136f02b8db48448907ae84cb9a687c, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/35d780aa28f64ba0a3a70e9aebecebc9, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/589a73044ba543aaa88615b5c7fa81ef, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/d852aa887d204a75a491acf54ea7df30, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/f803236f03614c06ac67f933c6df5f04, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/e0e6d189e2bf4a73bf565a69072640ae, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/0848704199784825a46410dd711b14da, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/a0883988b1074fa08b9e8bbbfff1b051, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/9ea82a92ae2b4abca98841eb3b399c24, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/882169c4d2b24fae8edf094dc9c4fedb, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/0c32e2f85ea340d29b8c54f84d7f5adc, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/4709bc6903fd458cba18f1d9dca9dc09, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/87647f51c71a43edb71d22fc6d788c48] to archive 2023-03-31 16:54:44,430 DEBUG [StoreCloser-TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-31 16:54:44,432 DEBUG [StoreCloser-TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/bb3f5d3f50584c91a07b0498a7d638eb to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/bb3f5d3f50584c91a07b0498a7d638eb 2023-03-31 16:54:44,433 DEBUG [StoreCloser-TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/9618f2346f0e4219afe0af8176a26ca9 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/9618f2346f0e4219afe0af8176a26ca9 2023-03-31 16:54:44,434 DEBUG [StoreCloser-TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/8ec4b187750e43ada713cc6e720ee32c to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/8ec4b187750e43ada713cc6e720ee32c 2023-03-31 16:54:44,435 DEBUG [StoreCloser-TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/cffe11007d554e74ba6f7b376f31e142 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/cffe11007d554e74ba6f7b376f31e142 2023-03-31 16:54:44,436 DEBUG [StoreCloser-TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/b87470db93dc46e7b06273abafeb16b9 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/b87470db93dc46e7b06273abafeb16b9 2023-03-31 16:54:44,437 DEBUG [StoreCloser-TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/074f475d70214880a9bbec74f8fc49b6 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/074f475d70214880a9bbec74f8fc49b6 2023-03-31 16:54:44,439 DEBUG [StoreCloser-TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/0fc1625fa8f74442bb50483deca7510c to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/0fc1625fa8f74442bb50483deca7510c 2023-03-31 16:54:44,440 DEBUG [StoreCloser-TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/b1c0303b69d445a3ad1ebd4679ebbd84 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/b1c0303b69d445a3ad1ebd4679ebbd84 2023-03-31 16:54:44,441 DEBUG [StoreCloser-TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/8239929e2581489190769f99bb68cbb3 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/8239929e2581489190769f99bb68cbb3 2023-03-31 16:54:44,442 DEBUG [StoreCloser-TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/7aec5dbcc7034377965cdb4265fe079d to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/7aec5dbcc7034377965cdb4265fe079d 2023-03-31 16:54:44,443 DEBUG [StoreCloser-TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/b21f40184b63425283a74bb12189bf64 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/b21f40184b63425283a74bb12189bf64 2023-03-31 16:54:44,444 DEBUG [StoreCloser-TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/e0e35469f2904e61bc0de0b657e04362 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/e0e35469f2904e61bc0de0b657e04362 2023-03-31 16:54:44,445 DEBUG [StoreCloser-TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/08fdc2d8e0ac4355bd611f9c1370ece5 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/08fdc2d8e0ac4355bd611f9c1370ece5 2023-03-31 16:54:44,446 DEBUG [StoreCloser-TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/1f136f02b8db48448907ae84cb9a687c to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/1f136f02b8db48448907ae84cb9a687c 2023-03-31 16:54:44,447 DEBUG [StoreCloser-TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/35d780aa28f64ba0a3a70e9aebecebc9 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/35d780aa28f64ba0a3a70e9aebecebc9 2023-03-31 16:54:44,448 DEBUG [StoreCloser-TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/589a73044ba543aaa88615b5c7fa81ef to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/589a73044ba543aaa88615b5c7fa81ef 2023-03-31 16:54:44,449 DEBUG [StoreCloser-TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/d852aa887d204a75a491acf54ea7df30 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/d852aa887d204a75a491acf54ea7df30 2023-03-31 16:54:44,450 DEBUG [StoreCloser-TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/f803236f03614c06ac67f933c6df5f04 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/f803236f03614c06ac67f933c6df5f04 2023-03-31 16:54:44,451 DEBUG [StoreCloser-TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/e0e6d189e2bf4a73bf565a69072640ae to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/e0e6d189e2bf4a73bf565a69072640ae 2023-03-31 16:54:44,452 DEBUG [StoreCloser-TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/0848704199784825a46410dd711b14da to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/0848704199784825a46410dd711b14da 2023-03-31 16:54:44,453 DEBUG [StoreCloser-TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/a0883988b1074fa08b9e8bbbfff1b051 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/a0883988b1074fa08b9e8bbbfff1b051 2023-03-31 16:54:44,454 DEBUG [StoreCloser-TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/9ea82a92ae2b4abca98841eb3b399c24 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/9ea82a92ae2b4abca98841eb3b399c24 2023-03-31 16:54:44,455 DEBUG [StoreCloser-TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/882169c4d2b24fae8edf094dc9c4fedb to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/882169c4d2b24fae8edf094dc9c4fedb 2023-03-31 16:54:44,456 DEBUG [StoreCloser-TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/0c32e2f85ea340d29b8c54f84d7f5adc to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/0c32e2f85ea340d29b8c54f84d7f5adc 2023-03-31 16:54:44,457 DEBUG [StoreCloser-TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/4709bc6903fd458cba18f1d9dca9dc09 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/4709bc6903fd458cba18f1d9dca9dc09 2023-03-31 16:54:44,457 DEBUG [StoreCloser-TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/87647f51c71a43edb71d22fc6d788c48 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/87647f51c71a43edb71d22fc6d788c48 2023-03-31 16:54:44,460 DEBUG [StoreCloser-TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/308dfc7bca3948e690cefe3994e99656, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/1582f5e7f9d246fa8594e5860f39e221, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/f7265421edeb4afab79c0e61a58898ca, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/0172887afb7a44b2bf66317ee3b84e47, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/329c493d8e7f4c79aa85852a3e5dcdd1, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/abd6b9ba3e694e18aeebe2c33a66fe92, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/4e399327d7954581853ef1d4ce813b37, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/644f985a1b5c41f1a493577b6829d5b0, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/ce3235521bee48aba86f8b30ae6cd971, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/be809c5544aa4162b2352f4a0b24f679, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/1b0e3918c6f04390b5dc193c87fee2aa, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/33c0856c380343b488cda9c1d784fe0d, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/a82e2b27555345279a58674ad41912ed, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/34b28e01df4347a3a3f785a681cbf748, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/06da64f690c84b029c942dac32b160b1, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/9bcd24273bb64a08a0a72f382aec503c, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/efe3fbbd80934e7e896de573c8cf8ddf, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/726aa24809b043d6adfb7cd24e6e6817, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/a96fa11ba27e4438af1889b2414ac68c, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/fdbd906ffb1e4b64906a7d7da8de7432, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/8d7ced1b6a2a4f4ba3d0a46063121ecc, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/5af78e50c5ac4422afdd16d5ba4c5543, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/9a94f5d5b3634b90b071c941e11083d2, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/9b06eafeadc54d21b9e48ab8369ddc1e, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/35a5b335b7694b9fa35e898b09dc0c03, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/9202765617f04e0da9cf06b5289d30c6] to archive 2023-03-31 16:54:44,461 DEBUG [StoreCloser-TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-31 16:54:44,462 DEBUG [StoreCloser-TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/308dfc7bca3948e690cefe3994e99656 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/308dfc7bca3948e690cefe3994e99656 2023-03-31 16:54:44,463 DEBUG [StoreCloser-TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/1582f5e7f9d246fa8594e5860f39e221 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/1582f5e7f9d246fa8594e5860f39e221 2023-03-31 16:54:44,464 DEBUG [StoreCloser-TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/f7265421edeb4afab79c0e61a58898ca to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/f7265421edeb4afab79c0e61a58898ca 2023-03-31 16:54:44,466 DEBUG [StoreCloser-TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/0172887afb7a44b2bf66317ee3b84e47 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/0172887afb7a44b2bf66317ee3b84e47 2023-03-31 16:54:44,467 DEBUG [StoreCloser-TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/329c493d8e7f4c79aa85852a3e5dcdd1 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/329c493d8e7f4c79aa85852a3e5dcdd1 2023-03-31 16:54:44,468 DEBUG [StoreCloser-TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/abd6b9ba3e694e18aeebe2c33a66fe92 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/abd6b9ba3e694e18aeebe2c33a66fe92 2023-03-31 16:54:44,469 DEBUG [StoreCloser-TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/4e399327d7954581853ef1d4ce813b37 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/4e399327d7954581853ef1d4ce813b37 2023-03-31 16:54:44,470 DEBUG [StoreCloser-TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/644f985a1b5c41f1a493577b6829d5b0 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/644f985a1b5c41f1a493577b6829d5b0 2023-03-31 16:54:44,471 DEBUG [StoreCloser-TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/ce3235521bee48aba86f8b30ae6cd971 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/ce3235521bee48aba86f8b30ae6cd971 2023-03-31 16:54:44,471 DEBUG [StoreCloser-TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/be809c5544aa4162b2352f4a0b24f679 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/be809c5544aa4162b2352f4a0b24f679 2023-03-31 16:54:44,472 DEBUG [StoreCloser-TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/1b0e3918c6f04390b5dc193c87fee2aa to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/1b0e3918c6f04390b5dc193c87fee2aa 2023-03-31 16:54:44,473 DEBUG [StoreCloser-TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/33c0856c380343b488cda9c1d784fe0d to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/33c0856c380343b488cda9c1d784fe0d 2023-03-31 16:54:44,475 DEBUG [StoreCloser-TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/a82e2b27555345279a58674ad41912ed to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/a82e2b27555345279a58674ad41912ed 2023-03-31 16:54:44,476 DEBUG [StoreCloser-TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/34b28e01df4347a3a3f785a681cbf748 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/34b28e01df4347a3a3f785a681cbf748 2023-03-31 16:54:44,477 DEBUG [StoreCloser-TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/06da64f690c84b029c942dac32b160b1 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/06da64f690c84b029c942dac32b160b1 2023-03-31 16:54:44,478 DEBUG [StoreCloser-TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/9bcd24273bb64a08a0a72f382aec503c to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/9bcd24273bb64a08a0a72f382aec503c 2023-03-31 16:54:44,479 DEBUG [StoreCloser-TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/efe3fbbd80934e7e896de573c8cf8ddf to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/efe3fbbd80934e7e896de573c8cf8ddf 2023-03-31 16:54:44,480 DEBUG [StoreCloser-TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/726aa24809b043d6adfb7cd24e6e6817 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/726aa24809b043d6adfb7cd24e6e6817 2023-03-31 16:54:44,481 DEBUG [StoreCloser-TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/a96fa11ba27e4438af1889b2414ac68c to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/a96fa11ba27e4438af1889b2414ac68c 2023-03-31 16:54:44,482 DEBUG [StoreCloser-TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/fdbd906ffb1e4b64906a7d7da8de7432 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/fdbd906ffb1e4b64906a7d7da8de7432 2023-03-31 16:54:44,483 DEBUG [StoreCloser-TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/8d7ced1b6a2a4f4ba3d0a46063121ecc to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/8d7ced1b6a2a4f4ba3d0a46063121ecc 2023-03-31 16:54:44,485 DEBUG [StoreCloser-TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/5af78e50c5ac4422afdd16d5ba4c5543 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/5af78e50c5ac4422afdd16d5ba4c5543 2023-03-31 16:54:44,485 DEBUG [StoreCloser-TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/9a94f5d5b3634b90b071c941e11083d2 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/9a94f5d5b3634b90b071c941e11083d2 2023-03-31 16:54:44,486 DEBUG [StoreCloser-TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/9b06eafeadc54d21b9e48ab8369ddc1e to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/9b06eafeadc54d21b9e48ab8369ddc1e 2023-03-31 16:54:44,487 DEBUG [StoreCloser-TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/35a5b335b7694b9fa35e898b09dc0c03 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/35a5b335b7694b9fa35e898b09dc0c03 2023-03-31 16:54:44,489 DEBUG [StoreCloser-TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/9202765617f04e0da9cf06b5289d30c6 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/9202765617f04e0da9cf06b5289d30c6 2023-03-31 16:54:44,493 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/recovered.edits/437.seqid, newMaxSeqId=437, maxSeqId=4 2023-03-31 16:54:44,495 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b. 2023-03-31 16:54:44,495 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for e76385ddbd37af190ba2df92b5602e3b: 2023-03-31 16:54:44,497 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed e76385ddbd37af190ba2df92b5602e3b 2023-03-31 16:54:44,497 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=e76385ddbd37af190ba2df92b5602e3b, regionState=CLOSED 2023-03-31 16:54:44,497 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680281684497"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680281684497"}]},"ts":"1680281684497"} 2023-03-31 16:54:44,499 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=25 2023-03-31 16:54:44,499 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=25, state=SUCCESS; CloseRegionProcedure e76385ddbd37af190ba2df92b5602e3b, server=jenkins-hbase4.apache.org,33441,1680281619375 in 1.5300 sec 2023-03-31 16:54:44,500 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=24 2023-03-31 16:54:44,500 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=24, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=e76385ddbd37af190ba2df92b5602e3b, UNASSIGN in 1.5340 sec 2023-03-31 16:54:44,501 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680281684501"}]},"ts":"1680281684501"} 2023-03-31 16:54:44,502 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-31 16:54:44,508 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-31 16:54:44,509 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=24, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.5510 sec 2023-03-31 16:54:45,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-31 16:54:45,065 INFO [Listener at localhost/44135] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 24 completed 2023-03-31 16:54:45,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-03-31 16:54:45,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure2.ProcedureExecutor(1029): Stored pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-31 16:54:45,067 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-31 16:54:45,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-03-31 16:54:45,068 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=27, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-31 16:54:45,073 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b 2023-03-31 16:54:45,076 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A, FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B, FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C, FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/recovered.edits] 2023-03-31 16:54:45,078 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/6b41ce19e46f4daca32517cbf2127b19 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/6b41ce19e46f4daca32517cbf2127b19 2023-03-31 16:54:45,079 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/757788f2184849f99ffa5d0ccf9771b3 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/757788f2184849f99ffa5d0ccf9771b3 2023-03-31 16:54:45,081 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/cc699c0c12774ebb9b669de5d317e0b8 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/A/cc699c0c12774ebb9b669de5d317e0b8 2023-03-31 16:54:45,083 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/cf988a9a42244d699dba69494ae0185c to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/cf988a9a42244d699dba69494ae0185c 2023-03-31 16:54:45,085 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/d652020d6643414fa94fa644a45449bb to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/d652020d6643414fa94fa644a45449bb 2023-03-31 16:54:45,086 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/fbbfa65fb92247fc8ea30a1b2313df3d to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/B/fbbfa65fb92247fc8ea30a1b2313df3d 2023-03-31 16:54:45,089 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/0fcbe793ed44473189dced508b71ef41 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/0fcbe793ed44473189dced508b71ef41 2023-03-31 16:54:45,090 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/8778daef02cc42c6b8c18820b4c185a9 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/8778daef02cc42c6b8c18820b4c185a9 2023-03-31 16:54:45,091 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/ec3565a623d049619c1d9a48d20495b9 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/C/ec3565a623d049619c1d9a48d20495b9 2023-03-31 16:54:45,095 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/recovered.edits/437.seqid to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b/recovered.edits/437.seqid 2023-03-31 16:54:45,095 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/e76385ddbd37af190ba2df92b5602e3b 2023-03-31 16:54:45,095 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-31 16:54:45,096 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-31 16:54:45,097 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-03-31 16:54:45,101 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033102ecdb9c58f946a5b0e75f97625ee49d to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033102ecdb9c58f946a5b0e75f97625ee49d 2023-03-31 16:54:45,102 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033103ba35a5fd3b41fea5fe7b8c1b53d74b to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033103ba35a5fd3b41fea5fe7b8c1b53d74b 2023-03-31 16:54:45,103 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033109589f253d734a2887fe7f3f34b294cb to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033109589f253d734a2887fe7f3f34b294cb 2023-03-31 16:54:45,105 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033114b0636d959a40f9b6ad97e7453d89c9 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033114b0636d959a40f9b6ad97e7453d89c9 2023-03-31 16:54:45,106 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033119aabb9ccfde4c8baf432f6076014a74 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033119aabb9ccfde4c8baf432f6076014a74 2023-03-31 16:54:45,108 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303312a8657c7e2be4bd7a957b94e55ecf000 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303312a8657c7e2be4bd7a957b94e55ecf000 2023-03-31 16:54:45,109 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303313322c635dc0f4fd9b108ead450f699c1 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303313322c635dc0f4fd9b108ead450f699c1 2023-03-31 16:54:45,110 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303314d7540a0dafa4cf59c4e0633a0d9ed62 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303314d7540a0dafa4cf59c4e0633a0d9ed62 2023-03-31 16:54:45,111 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033152562d917ff44df2aea9126dcdc4f6f4 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033152562d917ff44df2aea9126dcdc4f6f4 2023-03-31 16:54:45,113 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033152ea7abf49944c019d18b642fe18b295 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033152ea7abf49944c019d18b642fe18b295 2023-03-31 16:54:45,115 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303317aa5b56792e445979f8edf9984c6fdb7 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303317aa5b56792e445979f8edf9984c6fdb7 2023-03-31 16:54:45,116 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331819a5a6dec8849e49b1e88504f946199 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331819a5a6dec8849e49b1e88504f946199 2023-03-31 16:54:45,117 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303318adf260487eb47f6b4b3f5b6317af52c to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303318adf260487eb47f6b4b3f5b6317af52c 2023-03-31 16:54:45,118 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331907d95d7dd85492b983f6fb2016e63b4 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331907d95d7dd85492b983f6fb2016e63b4 2023-03-31 16:54:45,119 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303319656a47ed53a47d5a0771c4c2ea4ce3b to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303319656a47ed53a47d5a0771c4c2ea4ce3b 2023-03-31 16:54:45,120 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331ac9635d9d82f42ec8f02b43f67424353 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331ac9635d9d82f42ec8f02b43f67424353 2023-03-31 16:54:45,121 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331c5139450d6e2450eb9b2506c301e836f to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331c5139450d6e2450eb9b2506c301e836f 2023-03-31 16:54:45,122 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331cd717c6f0dbe40a0afd249730471ab01 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331cd717c6f0dbe40a0afd249730471ab01 2023-03-31 16:54:45,124 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331daaffc4adbdc49e3b576157b43f4855e to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331daaffc4adbdc49e3b576157b43f4855e 2023-03-31 16:54:45,125 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331e67ff55bf1fb4b2da18958d4d8f1df08 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331e67ff55bf1fb4b2da18958d4d8f1df08 2023-03-31 16:54:45,126 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331ed1519f576a94a9f8b7866a74c5f2475 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331ed1519f576a94a9f8b7866a74c5f2475 2023-03-31 16:54:45,127 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331f68a2a0c90244d46aa364f2760dbbb09 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331f68a2a0c90244d46aa364f2760dbbb09 2023-03-31 16:54:45,129 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331fc57b62b5069462491749dbb371249d5 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331fc57b62b5069462491749dbb371249d5 2023-03-31 16:54:45,129 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-31 16:54:45,132 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=27, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-31 16:54:45,134 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-31 16:54:45,136 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-31 16:54:45,137 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=27, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-31 16:54:45,137 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-31 16:54:45,137 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680281685137"}]},"ts":"9223372036854775807"} 2023-03-31 16:54:45,139 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-31 16:54:45,139 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => e76385ddbd37af190ba2df92b5602e3b, NAME => 'TestAcidGuarantees,,1680281650799.e76385ddbd37af190ba2df92b5602e3b.', STARTKEY => '', ENDKEY => ''}] 2023-03-31 16:54:45,139 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-31 16:54:45,139 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680281685139"}]},"ts":"9223372036854775807"} 2023-03-31 16:54:45,141 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-31 16:54:45,142 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=27, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-31 16:54:45,143 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=27, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 77 msec 2023-03-31 16:54:45,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-03-31 16:54:45,169 INFO [Listener at localhost/44135] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 27 completed 2023-03-31 16:54:45,181 INFO [Listener at localhost/44135] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=224 (was 232), OpenFileDescriptor=511 (was 581), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=341 (was 371), ProcessCount=169 (was 169), AvailableMemoryMB=3730 (was 3825) 2023-03-31 16:54:45,192 INFO [Listener at localhost/44135] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=224, OpenFileDescriptor=511, MaxFileDescriptor=60000, SystemLoadAverage=341, ProcessCount=169, AvailableMemoryMB=3730 2023-03-31 16:54:45,194 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-31 16:54:45,195 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-31 16:54:45,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure2.ProcedureExecutor(1029): Stored pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-31 16:54:45,197 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-31 16:54:45,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 28 2023-03-31 16:54:45,198 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-31 16:54:45,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-31 16:54:45,199 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8 2023-03-31 16:54:45,200 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8 empty. 2023-03-31 16:54:45,200 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8 2023-03-31 16:54:45,200 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-31 16:54:45,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-31 16:54:45,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-31 16:54:45,610 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-31 16:54:45,612 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 3ab680f7c89c77c9be754a4581df5bc8, NAME => 'TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp 2023-03-31 16:54:45,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-31 16:54:46,018 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-31 16:54:46,018 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 3ab680f7c89c77c9be754a4581df5bc8, disabling compactions & flushes 2023-03-31 16:54:46,018 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:46,018 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:46,018 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. after waiting 0 ms 2023-03-31 16:54:46,018 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:46,018 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:46,018 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 3ab680f7c89c77c9be754a4581df5bc8: 2023-03-31 16:54:46,020 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-31 16:54:46,021 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680281686021"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680281686021"}]},"ts":"1680281686021"} 2023-03-31 16:54:46,022 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-31 16:54:46,022 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-31 16:54:46,023 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680281686023"}]},"ts":"1680281686023"} 2023-03-31 16:54:46,023 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-31 16:54:46,027 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=3ab680f7c89c77c9be754a4581df5bc8, ASSIGN}] 2023-03-31 16:54:46,029 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=3ab680f7c89c77c9be754a4581df5bc8, ASSIGN 2023-03-31 16:54:46,029 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=3ab680f7c89c77c9be754a4581df5bc8, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,33441,1680281619375; forceNewPlan=false, retain=false 2023-03-31 16:54:46,180 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=3ab680f7c89c77c9be754a4581df5bc8, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:46,181 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680281686180"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680281686180"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680281686180"}]},"ts":"1680281686180"} 2023-03-31 16:54:46,182 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=29, state=RUNNABLE; OpenRegionProcedure 3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375}] 2023-03-31 16:54:46,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-31 16:54:46,337 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:46,337 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 3ab680f7c89c77c9be754a4581df5bc8, NAME => 'TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.', STARTKEY => '', ENDKEY => ''} 2023-03-31 16:54:46,337 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 3ab680f7c89c77c9be754a4581df5bc8 2023-03-31 16:54:46,338 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-31 16:54:46,338 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 3ab680f7c89c77c9be754a4581df5bc8 2023-03-31 16:54:46,338 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 3ab680f7c89c77c9be754a4581df5bc8 2023-03-31 16:54:46,339 INFO [StoreOpener-3ab680f7c89c77c9be754a4581df5bc8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 3ab680f7c89c77c9be754a4581df5bc8 2023-03-31 16:54:46,341 DEBUG [StoreOpener-3ab680f7c89c77c9be754a4581df5bc8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A 2023-03-31 16:54:46,341 DEBUG [StoreOpener-3ab680f7c89c77c9be754a4581df5bc8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A 2023-03-31 16:54:46,341 INFO [StoreOpener-3ab680f7c89c77c9be754a4581df5bc8-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-31 16:54:46,341 INFO [StoreOpener-3ab680f7c89c77c9be754a4581df5bc8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3ab680f7c89c77c9be754a4581df5bc8 columnFamilyName A 2023-03-31 16:54:46,342 INFO [StoreOpener-3ab680f7c89c77c9be754a4581df5bc8-1] regionserver.HStore(310): Store=3ab680f7c89c77c9be754a4581df5bc8/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-31 16:54:46,342 INFO [StoreOpener-3ab680f7c89c77c9be754a4581df5bc8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 3ab680f7c89c77c9be754a4581df5bc8 2023-03-31 16:54:46,343 DEBUG [StoreOpener-3ab680f7c89c77c9be754a4581df5bc8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B 2023-03-31 16:54:46,343 DEBUG [StoreOpener-3ab680f7c89c77c9be754a4581df5bc8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B 2023-03-31 16:54:46,343 INFO [StoreOpener-3ab680f7c89c77c9be754a4581df5bc8-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-31 16:54:46,343 INFO [StoreOpener-3ab680f7c89c77c9be754a4581df5bc8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3ab680f7c89c77c9be754a4581df5bc8 columnFamilyName B 2023-03-31 16:54:46,344 INFO [StoreOpener-3ab680f7c89c77c9be754a4581df5bc8-1] regionserver.HStore(310): Store=3ab680f7c89c77c9be754a4581df5bc8/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-31 16:54:46,344 INFO [StoreOpener-3ab680f7c89c77c9be754a4581df5bc8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 3ab680f7c89c77c9be754a4581df5bc8 2023-03-31 16:54:46,345 DEBUG [StoreOpener-3ab680f7c89c77c9be754a4581df5bc8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C 2023-03-31 16:54:46,345 DEBUG [StoreOpener-3ab680f7c89c77c9be754a4581df5bc8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C 2023-03-31 16:54:46,345 INFO [StoreOpener-3ab680f7c89c77c9be754a4581df5bc8-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-31 16:54:46,345 INFO [StoreOpener-3ab680f7c89c77c9be754a4581df5bc8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3ab680f7c89c77c9be754a4581df5bc8 columnFamilyName C 2023-03-31 16:54:46,345 INFO [StoreOpener-3ab680f7c89c77c9be754a4581df5bc8-1] regionserver.HStore(310): Store=3ab680f7c89c77c9be754a4581df5bc8/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-31 16:54:46,346 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:46,346 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8 2023-03-31 16:54:46,347 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8 2023-03-31 16:54:46,349 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-31 16:54:46,350 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 3ab680f7c89c77c9be754a4581df5bc8 2023-03-31 16:54:46,351 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-31 16:54:46,352 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 3ab680f7c89c77c9be754a4581df5bc8; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64896296, jitterRate=-0.03296983242034912}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-31 16:54:46,352 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 3ab680f7c89c77c9be754a4581df5bc8: 2023-03-31 16:54:46,353 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8., pid=30, masterSystemTime=1680281686333 2023-03-31 16:54:46,354 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:46,354 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:46,354 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=3ab680f7c89c77c9be754a4581df5bc8, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:46,354 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680281686354"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680281686354"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680281686354"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680281686354"}]},"ts":"1680281686354"} 2023-03-31 16:54:46,357 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=29 2023-03-31 16:54:46,357 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=29, state=SUCCESS; OpenRegionProcedure 3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 in 173 msec 2023-03-31 16:54:46,359 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=28 2023-03-31 16:54:46,359 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=3ab680f7c89c77c9be754a4581df5bc8, ASSIGN in 330 msec 2023-03-31 16:54:46,359 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-31 16:54:46,359 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680281686359"}]},"ts":"1680281686359"} 2023-03-31 16:54:46,360 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-31 16:54:46,362 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-31 16:54:46,364 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=28, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1680 sec 2023-03-31 16:54:47,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-31 16:54:47,303 INFO [Listener at localhost/44135] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 28 completed 2023-03-31 16:54:47,303 DEBUG [Listener at localhost/44135] zookeeper.ReadOnlyZKClient(139): Connect 0x22b4eae5 to 127.0.0.1:51916 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 16:54:47,308 DEBUG [Listener at localhost/44135] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@12e82edd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 16:54:47,309 DEBUG [Listener at localhost/44135] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 16:54:47,310 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40700, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 16:54:47,311 DEBUG [Listener at localhost/44135] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-31 16:54:47,313 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56606, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-31 16:54:47,314 DEBUG [Listener at localhost/44135] zookeeper.ReadOnlyZKClient(139): Connect 0x262f4570 to 127.0.0.1:51916 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 16:54:47,318 DEBUG [Listener at localhost/44135] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@23c97092, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 16:54:47,318 DEBUG [Listener at localhost/44135] zookeeper.ReadOnlyZKClient(139): Connect 0x00ca898c to 127.0.0.1:51916 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 16:54:47,322 DEBUG [Listener at localhost/44135] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5c124820, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 16:54:47,322 DEBUG [Listener at localhost/44135] zookeeper.ReadOnlyZKClient(139): Connect 0x6585a70a to 127.0.0.1:51916 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 16:54:47,327 DEBUG [Listener at localhost/44135] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3157d4df, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 16:54:47,327 DEBUG [Listener at localhost/44135] zookeeper.ReadOnlyZKClient(139): Connect 0x7480f9da to 127.0.0.1:51916 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 16:54:47,331 DEBUG [Listener at localhost/44135] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3cf5bdb7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 16:54:47,331 DEBUG [Listener at localhost/44135] zookeeper.ReadOnlyZKClient(139): Connect 0x1a24d617 to 127.0.0.1:51916 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 16:54:47,334 DEBUG [Listener at localhost/44135] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6919b2a0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 16:54:47,334 DEBUG [Listener at localhost/44135] zookeeper.ReadOnlyZKClient(139): Connect 0x2ce5573e to 127.0.0.1:51916 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 16:54:47,338 DEBUG [Listener at localhost/44135] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@8cd7cb0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 16:54:47,339 DEBUG [Listener at localhost/44135] zookeeper.ReadOnlyZKClient(139): Connect 0x087b3095 to 127.0.0.1:51916 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 16:54:47,345 DEBUG [Listener at localhost/44135] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@ea1ca03, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 16:54:47,345 DEBUG [Listener at localhost/44135] zookeeper.ReadOnlyZKClient(139): Connect 0x12115052 to 127.0.0.1:51916 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 16:54:47,349 DEBUG [Listener at localhost/44135] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@433f251d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 16:54:47,349 DEBUG [Listener at localhost/44135] zookeeper.ReadOnlyZKClient(139): Connect 0x7e65578e to 127.0.0.1:51916 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 16:54:47,354 DEBUG [Listener at localhost/44135] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@66ee0bd0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 16:54:47,355 DEBUG [Listener at localhost/44135] zookeeper.ReadOnlyZKClient(139): Connect 0x6fcbaba8 to 127.0.0.1:51916 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 16:54:47,358 DEBUG [Listener at localhost/44135] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3b4658f8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 16:54:47,361 DEBUG [hconnection-0x7f6ebbca-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 16:54:47,362 DEBUG [hconnection-0x9646119-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 16:54:47,362 DEBUG [hconnection-0x522a5711-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 16:54:47,368 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40716, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 16:54:47,369 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40732, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 16:54:47,368 DEBUG [hconnection-0x653972e0-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 16:54:47,368 DEBUG [hconnection-0x74ae6e64-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 16:54:47,368 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40720, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 16:54:47,369 DEBUG [hconnection-0x5571e5f6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 16:54:47,369 DEBUG [hconnection-0x6793e347-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 16:54:47,369 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:47,370 DEBUG [hconnection-0x704198d2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 16:54:47,369 DEBUG [hconnection-0x2cea50c2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 16:54:47,370 DEBUG [hconnection-0x3aff5eb7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 16:54:47,377 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40750, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 16:54:47,377 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40748, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 16:54:47,377 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40766, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 16:54:47,377 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40782, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 16:54:47,377 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40768, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 16:54:47,378 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40790, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 16:54:47,378 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40794, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 16:54:47,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:47,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:47,380 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:47,380 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:47,380 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:47,380 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:47,381 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:47,381 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:47,387 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:47,387 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:47,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:47,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:47,389 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:47,389 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:47,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:47,389 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:47,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:47,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:47,390 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:47,390 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:47,390 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:47,390 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:47,390 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:47,391 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:47,391 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:47,391 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:47,391 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. started... 2023-03-31 16:54:47,391 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3ab680f7c89c77c9be754a4581df5bc8 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-03-31 16:54:47,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on 3ab680f7c89c77c9be754a4581df5bc8 2023-03-31 16:54:47,393 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=A 2023-03-31 16:54:47,393 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:47,393 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=B 2023-03-31 16:54:47,393 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:47,393 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=C 2023-03-31 16:54:47,393 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:47,395 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. as already flushing 2023-03-31 16:54:47,415 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680281687404, max timestamp=1680281687412Num uniques 859; 2023-03-31 16:54:47,415 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680281687404, max timestamp=1680281687412Num uniques 859; 2023-03-31 16:54:47,415 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 16:54:47,415 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 16:54:47,417 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680281687404, max timestamp=1680281687412Num uniques 859; 2023-03-31 16:54:47,417 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 16:54:47,438 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:47,438 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:47,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40794 deadline: 1680281747433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:47,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40748 deadline: 1680281747433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:47,438 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:47,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40768 deadline: 1680281747434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:47,440 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680281687425, max timestamp=1680281687437Num uniques 1018; 2023-03-31 16:54:47,440 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680281687425, max timestamp=1680281687437Num uniques 1018; 2023-03-31 16:54:47,440 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-03-31 16:54:47,440 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680281687425, max timestamp=1680281687437Num uniques 1018; 2023-03-31 16:54:47,440 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=6870, suffix heap size=133840,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=150 2023-03-31 16:54:47,441 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-03-31 16:54:47,467 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:47,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40750 deadline: 1680281747465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:47,469 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:47,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40716 deadline: 1680281747467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:47,542 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:47,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40748 deadline: 1680281747541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:47,543 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:47,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40794 deadline: 1680281747541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:47,544 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:47,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40768 deadline: 1680281747542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:47,569 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:47,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40750 deadline: 1680281747568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:47,572 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:47,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40716 deadline: 1680281747571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:47,747 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:47,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40748 deadline: 1680281747745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:47,747 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:47,748 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:47,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40794 deadline: 1680281747745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:47,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40768 deadline: 1680281747745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:47,772 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:47,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40750 deadline: 1680281747770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:47,774 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:47,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40716 deadline: 1680281747773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:47,844 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=11 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/A/c682f0805d12414c8ec7285708401af1 2023-03-31 16:54:48,049 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:48,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40748 deadline: 1680281748048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:48,051 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:48,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40794 deadline: 1680281748050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:48,051 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:48,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40768 deadline: 1680281748050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:48,077 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:48,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40750 deadline: 1680281748075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:48,077 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:48,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40716 deadline: 1680281748076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:48,257 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=11 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/B/892607cbf9fb41b8b3c8688d5135ff94 2023-03-31 16:54:48,268 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=11 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/C/8a23d40f8d7c46a48e3e5f02e537f0c0 2023-03-31 16:54:48,272 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/A/c682f0805d12414c8ec7285708401af1 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/c682f0805d12414c8ec7285708401af1 2023-03-31 16:54:48,277 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/c682f0805d12414c8ec7285708401af1, entries=150, sequenceid=11, filesize=11.6 K 2023-03-31 16:54:48,278 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/B/892607cbf9fb41b8b3c8688d5135ff94 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/892607cbf9fb41b8b3c8688d5135ff94 2023-03-31 16:54:48,281 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/892607cbf9fb41b8b3c8688d5135ff94, entries=150, sequenceid=11, filesize=11.6 K 2023-03-31 16:54:48,282 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/C/8a23d40f8d7c46a48e3e5f02e537f0c0 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/8a23d40f8d7c46a48e3e5f02e537f0c0 2023-03-31 16:54:48,293 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/8a23d40f8d7c46a48e3e5f02e537f0c0, entries=150, sequenceid=11, filesize=11.6 K 2023-03-31 16:54:48,294 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=131.14 KB/134292 for 3ab680f7c89c77c9be754a4581df5bc8 in 903ms, sequenceid=11, compaction requested=false 2023-03-31 16:54:48,294 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3ab680f7c89c77c9be754a4581df5bc8: 2023-03-31 16:54:48,294 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:48,294 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:48,294 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:48,294 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:48,294 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:48,295 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:48,296 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:48,296 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:48,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:48,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:48,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:48,296 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:48,296 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:48,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:48,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:48,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:48,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:48,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:48,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:48,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@606f91b3[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:48,298 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:48,298 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:48,302 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:48,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:48,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:48,302 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:48,302 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:48,302 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:48,302 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:48,302 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:48,303 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:48,303 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:48,303 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:48,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:48,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:48,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:48,303 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:48,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:48,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:48,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:48,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:48,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:48,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:48,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:48,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:48,306 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:48,306 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:48,306 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:48,306 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:48,306 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:48,306 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:48,307 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:48,307 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:48,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:48,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:48,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:48,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:48,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:48,307 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:48,308 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:48,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:48,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:48,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:48,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:48,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:48,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:48,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:48,309 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:48,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:48,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:48,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:48,313 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:48,313 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:48,313 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:48,313 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:48,313 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:48,313 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:48,313 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:48,313 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:48,313 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:48,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:48,313 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:48,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:48,313 DEBUG [Thread-1666] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:48,313 DEBUG [Thread-1666] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:48,313 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:48,313 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:48,313 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:48,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:48,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:48,413 DEBUG [Thread-1666] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:48,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:48,416 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:48,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:48,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:48,417 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:48,417 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:48,417 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:48,418 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:48,418 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:48,418 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:48,419 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:48,419 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:48,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:48,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:48,419 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:48,419 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:48,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:48,420 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:48,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:48,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:48,420 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:48,420 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:48,420 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:48,421 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:48,421 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:48,421 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:48,421 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:48,421 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:48,421 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:48,421 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. started... 2023-03-31 16:54:48,421 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3ab680f7c89c77c9be754a4581df5bc8 3/3 column families, dataSize=131.14 KB heapSize=372.55 KB 2023-03-31 16:54:48,421 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=A 2023-03-31 16:54:48,421 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:54:48,422 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=B 2023-03-31 16:54:48,422 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:54:48,422 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=C 2023-03-31 16:54:48,422 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:54:48,429 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=64 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/A/0c73b5d47d0645778db68c0c3a116615 2023-03-31 16:54:48,441 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=64 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/B/3625c7f038f64ec6a85f794a2e6e0f84 2023-03-31 16:54:48,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on 3ab680f7c89c77c9be754a4581df5bc8 2023-03-31 16:54:48,555 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. as already flushing 2023-03-31 16:54:48,565 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:48,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40794 deadline: 1680281748563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:48,565 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:48,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40748 deadline: 1680281748563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:48,566 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:48,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40768 deadline: 1680281748564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:48,582 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:48,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40750 deadline: 1680281748580, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:48,582 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:48,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40716 deadline: 1680281748580, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:48,667 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:48,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40794 deadline: 1680281748666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:48,667 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:48,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40748 deadline: 1680281748666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:48,668 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:48,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40768 deadline: 1680281748667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:48,851 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=41.69 KB at sequenceid=64 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/C/e5b3f259af94447a8e2f280b1d9ad17e 2023-03-31 16:54:48,855 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/A/0c73b5d47d0645778db68c0c3a116615 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/0c73b5d47d0645778db68c0c3a116615 2023-03-31 16:54:48,859 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/0c73b5d47d0645778db68c0c3a116615, entries=150, sequenceid=64, filesize=11.6 K 2023-03-31 16:54:48,860 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/B/3625c7f038f64ec6a85f794a2e6e0f84 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/3625c7f038f64ec6a85f794a2e6e0f84 2023-03-31 16:54:48,865 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/3625c7f038f64ec6a85f794a2e6e0f84, entries=150, sequenceid=64, filesize=11.6 K 2023-03-31 16:54:48,866 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/C/e5b3f259af94447a8e2f280b1d9ad17e as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/e5b3f259af94447a8e2f280b1d9ad17e 2023-03-31 16:54:48,870 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:48,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40748 deadline: 1680281748869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:48,871 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/e5b3f259af94447a8e2f280b1d9ad17e, entries=150, sequenceid=64, filesize=11.6 K 2023-03-31 16:54:48,872 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~131.14 KB/134292, heapSize ~372.50 KB/381440, currentSize=53.67 KB/54960 for 3ab680f7c89c77c9be754a4581df5bc8 in 451ms, sequenceid=64, compaction requested=false 2023-03-31 16:54:48,872 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3ab680f7c89c77c9be754a4581df5bc8: 2023-03-31 16:54:48,872 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:48,872 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:48,872 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:48,872 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:48,872 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:48,872 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:48,874 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:48,874 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:48,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:48,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:48,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:48,874 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:48,874 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:48,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:48,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:48,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:48,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:48,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:48,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:48,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47ad75a5[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:48,876 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:48,876 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:48,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on 3ab680f7c89c77c9be754a4581df5bc8 2023-03-31 16:54:48,876 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3ab680f7c89c77c9be754a4581df5bc8 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-31 16:54:48,878 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:48,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:48,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:48,878 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:48,878 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:48,878 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:48,879 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:48,879 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:48,880 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:48,880 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:48,880 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=A 2023-03-31 16:54:48,880 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:48,880 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=B 2023-03-31 16:54:48,880 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:48,880 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:48,880 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:48,880 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=C 2023-03-31 16:54:48,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:48,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:48,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:48,880 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:48,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:48,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:48,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:48,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:48,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:48,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:48,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:48,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:48,885 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:48,885 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:48,885 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:48,885 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:48,885 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:48,885 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:48,886 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:48,887 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:48,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:48,887 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:48,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:48,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:48,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:48,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:48,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:48,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:48,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:48,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:48,888 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:48,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:48,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:48,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:48,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:48,890 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:48,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:48,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:48,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:48,893 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:48,893 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:48,893 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:48,893 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:48,893 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:48,893 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:48,893 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:48,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:48,894 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:48,894 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:48,894 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:48,894 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:48,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:48,894 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:48,894 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:48,894 DEBUG [Thread-1666] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:48,894 DEBUG [Thread-1666] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:48,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:48,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:48,915 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:48,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40768 deadline: 1680281748911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:48,918 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:48,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40794 deadline: 1680281748915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:48,994 DEBUG [Thread-1666] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:48,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:48,996 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:48,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:48,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:48,997 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:48,997 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:48,998 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:48,998 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:48,998 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:48,998 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:49,000 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,000 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:49,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:49,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:49,000 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,000 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:49,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:49,001 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:49,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:49,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:49,001 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:49,001 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:49,001 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:49,001 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:49,001 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:49,001 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:49,001 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:49,002 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:49,002 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:49,002 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. started... 2023-03-31 16:54:49,002 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. as already flushing 2023-03-31 16:54:49,002 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:49,002 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:49,002 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:49,002 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,002 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:49,002 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:49,004 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,004 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:49,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:49,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:49,004 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:49,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:49,004 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:49,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:49,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:49,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:49,005 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:49,005 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@683d1623[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:49,005 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:49,005 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:49,006 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:49,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:49,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:49,006 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:49,006 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:49,006 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:49,006 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,007 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:49,007 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:49,007 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,008 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:49,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:49,008 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:49,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:49,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:49,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:49,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:49,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:49,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:49,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:49,010 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:49,010 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:49,010 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:49,010 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:49,010 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:49,010 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:49,013 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:49,013 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:49,013 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:49,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:49,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:49,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:49,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:49,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:49,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:49,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:49,014 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:49,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:49,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:49,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:49,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:49,015 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:49,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:49,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,017 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:49,017 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:49,017 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:49,017 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:49,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:49,017 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:49,018 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:49,018 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,018 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:49,018 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:49,018 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:49,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:49,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:49,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:49,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:49,018 DEBUG [Thread-1666] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:49,018 DEBUG [Thread-1666] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:49,020 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:49,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40768 deadline: 1680281749018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,020 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:49,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40794 deadline: 1680281749019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,118 DEBUG [Thread-1666] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:49,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:49,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:49,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:49,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:49,122 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:49,122 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:49,122 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:49,122 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:49,122 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:49,122 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:49,124 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:49,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:49,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:49,124 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:49,124 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,124 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:49,124 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:49,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:49,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:49,125 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:49,125 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:49,125 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:49,125 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:49,125 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:49,125 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:49,125 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:49,125 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:49,125 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:49,126 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. started... 2023-03-31 16:54:49,126 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. as already flushing 2023-03-31 16:54:49,126 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:49,126 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:49,126 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:49,126 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,126 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:49,126 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:49,127 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,127 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:49,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:49,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:49,128 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:49,128 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:49,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:49,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:49,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:49,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:49,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:49,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63e674f0[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:49,129 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:49,129 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:49,131 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:49,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:49,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:49,131 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:49,131 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:49,131 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:49,131 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,131 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:49,135 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:49,135 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,135 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:49,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:49,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:49,135 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:49,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:49,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:49,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:49,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:49,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:49,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:49,139 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:49,139 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:49,139 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:49,139 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:49,139 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:49,139 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:49,140 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:49,140 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:49,140 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:49,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:49,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:49,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:49,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:49,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:49,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:49,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:49,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:49,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:49,141 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:49,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:49,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:49,142 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:49,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:49,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,145 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:49,145 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:49,145 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,145 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:49,145 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:49,145 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,145 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:49,145 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:49,146 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:49,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:49,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:49,146 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:49,145 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:49,146 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:49,145 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:49,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:49,146 DEBUG [Thread-1666] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:49,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:49,146 DEBUG [Thread-1666] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:49,175 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:49,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40748 deadline: 1680281749173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,223 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:49,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40768 deadline: 1680281749222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,225 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:49,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40794 deadline: 1680281749222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,246 DEBUG [Thread-1666] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:49,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:49,251 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:49,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:49,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:49,252 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:49,252 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:49,252 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:49,252 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:49,252 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:49,253 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:49,255 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:49,255 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:49,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:49,256 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,256 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:49,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:49,256 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:49,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:49,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:49,258 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:49,260 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:49,260 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:49,260 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:49,260 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:49,260 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:49,268 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:49,268 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:49,268 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:49,268 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. started... 2023-03-31 16:54:49,268 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. as already flushing 2023-03-31 16:54:49,268 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:49,268 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:49,268 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:49,268 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,268 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:49,268 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:49,269 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,269 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:49,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:49,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:49,270 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:49,270 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:49,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:49,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:49,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:49,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:49,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:49,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f989336[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:49,271 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:49,271 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:49,273 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:49,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:49,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:49,273 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:49,273 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,273 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:49,273 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:49,273 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:49,275 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,275 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:49,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:49,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:49,275 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:49,275 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:49,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:49,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:49,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:49,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:49,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:49,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:49,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:49,280 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:49,280 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:49,280 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:49,281 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:49,281 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:49,283 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:49,283 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:49,283 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:49,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:49,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:49,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:49,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:49,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:49,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:49,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:49,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:49,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:49,285 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:49,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:49,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:49,286 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:49,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:49,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,289 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:49,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:49,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:49,289 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,290 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:49,290 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:49,290 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,290 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:49,290 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:49,290 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:49,290 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:49,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:49,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:49,290 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:49,290 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:49,290 DEBUG [Thread-1666] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:49,290 DEBUG [Thread-1666] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:49,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:49,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:49,327 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=77 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/A/4a936689ba9243df8d88098f23782fce 2023-03-31 16:54:49,390 DEBUG [Thread-1666] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:49,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:49,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:49,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:49,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:49,404 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:49,404 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:49,404 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:49,404 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:49,405 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:49,405 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:49,406 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:49,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:49,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:49,406 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,407 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:49,407 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:49,408 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:49,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:49,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:49,409 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:49,412 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:49,412 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:49,412 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:49,412 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:49,412 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:49,416 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:49,416 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:49,416 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:49,416 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. started... 2023-03-31 16:54:49,416 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. as already flushing 2023-03-31 16:54:49,416 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:49,416 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:49,416 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:49,416 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,416 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:49,416 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:49,418 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:49,418 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:49,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:49,419 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:49,419 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:49,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:49,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:49,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:49,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:49,420 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:49,420 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53b4d13d[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:49,421 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:49,421 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:49,423 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:49,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:49,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:49,423 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:49,423 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,423 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:49,423 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:49,424 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:49,426 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,426 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:49,426 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:49,426 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:49,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:49,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:49,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:49,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:49,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:49,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:49,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:49,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:49,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:49,435 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:49,435 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:49,435 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:49,436 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:49,436 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:49,438 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:49,438 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:49,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:49,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:49,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:49,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:49,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:49,438 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:49,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:49,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:49,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:49,439 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:49,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:49,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:49,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:49,441 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:49,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:49,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,444 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,444 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:49,444 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:49,444 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:49,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:49,444 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:49,444 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:49,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:49,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:49,444 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:49,445 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,445 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:49,445 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:49,445 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:49,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:49,445 DEBUG [Thread-1666] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:49,445 DEBUG [Thread-1666] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:49,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:49,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:49,528 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:49,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40768 deadline: 1680281749526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,529 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:49,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40794 deadline: 1680281749527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,545 DEBUG [Thread-1666] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:49,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:49,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:49,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:49,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:49,549 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:49,549 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:49,549 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:49,549 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:49,549 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:49,549 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:49,552 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:49,552 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:49,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:49,552 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,552 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:49,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:49,552 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:49,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:49,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:49,553 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:49,553 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:49,553 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:49,553 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:49,553 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:49,553 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:49,553 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:49,553 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:49,553 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:49,554 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. started... 2023-03-31 16:54:49,554 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. as already flushing 2023-03-31 16:54:49,554 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:49,554 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:49,554 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:49,554 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,554 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:49,554 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:49,556 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,556 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:49,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:49,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:49,556 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:49,556 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:49,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:49,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:49,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:49,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:49,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:49,557 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:49,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53384966[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:49,558 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:49,559 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:49,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:49,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:49,559 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:49,559 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:49,560 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:49,559 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,560 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:49,561 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,561 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:49,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:49,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:49,561 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:49,561 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:49,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:49,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:49,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:49,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:49,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:49,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:49,563 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:49,563 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:49,563 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:49,563 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:49,563 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:49,564 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:49,566 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:49,566 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:49,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:49,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:49,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:49,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:49,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:49,566 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:49,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:49,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:49,566 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:49,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:49,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:49,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:49,568 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:49,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:49,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:49,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,577 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:49,577 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:49,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:49,577 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:49,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:49,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:49,578 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:49,577 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:49,578 DEBUG [Thread-1666] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:49,577 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:49,578 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:49,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:49,578 DEBUG [Thread-1666] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:49,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:49,578 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,578 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:49,578 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:49,578 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:49,592 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:49,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40716 deadline: 1680281749591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,593 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:49,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40750 deadline: 1680281749592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,678 DEBUG [Thread-1666] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:49,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:49,680 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:49,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40748 deadline: 1680281749679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:49,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:49,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:49,681 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:49,681 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:49,681 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:49,681 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:49,682 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:49,682 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:49,683 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:49,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:49,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:49,683 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:49,684 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,684 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:49,684 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:49,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:49,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:49,685 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:49,685 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:49,685 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:49,685 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:49,685 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:49,685 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:49,685 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:49,685 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:49,685 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:49,685 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. started... 2023-03-31 16:54:49,686 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. as already flushing 2023-03-31 16:54:49,686 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:49,686 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:49,686 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:49,686 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,686 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:49,686 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:49,687 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:49,687 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:49,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:49,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:49,688 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:49,688 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:49,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:49,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:49,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:49,690 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:49,690 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@329e605[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:49,690 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:49,690 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:49,692 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:49,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:49,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:49,693 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:49,693 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,693 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:49,693 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:49,693 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:49,694 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:49,694 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:49,694 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,695 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:49,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:49,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:49,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:49,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:49,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:49,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:49,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:49,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:49,698 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:49,698 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:49,698 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:49,698 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:49,698 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:49,698 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:49,699 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:49,699 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:49,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:49,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:49,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:49,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:49,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:49,699 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:49,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:49,700 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:49,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:49,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:49,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:49,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:49,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:49,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,702 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:49,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:49,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,705 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:49,705 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:49,705 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:49,705 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:49,706 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:49,706 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:49,706 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,706 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:49,706 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:49,705 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:49,705 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:49,706 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:49,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:49,706 DEBUG [Thread-1666] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:49,706 DEBUG [Thread-1666] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:49,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:49,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:49,737 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=77 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/B/4005fb371227416ab321e650f03677de 2023-03-31 16:54:49,806 DEBUG [Thread-1666] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:49,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:49,809 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:49,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:49,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:49,810 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:49,810 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:49,810 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:49,810 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:49,810 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:49,810 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:49,811 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:49,811 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:49,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:49,812 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,812 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:49,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:49,812 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:49,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:49,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:49,813 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:49,813 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:49,813 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:49,813 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:49,813 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:49,813 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:49,813 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:49,813 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:49,813 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:49,813 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. started... 2023-03-31 16:54:49,813 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. as already flushing 2023-03-31 16:54:49,813 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:49,814 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:49,814 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:49,814 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,814 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:49,814 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:49,815 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:49,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:49,815 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:49,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:49,815 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:49,816 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:49,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:49,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:49,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:49,817 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:49,817 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3813d475[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:49,817 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:49,817 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:49,819 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:49,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:49,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:49,819 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:49,819 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,819 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:49,819 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:49,819 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:49,821 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,821 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:49,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:49,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:49,821 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:49,821 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:49,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:49,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:49,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:49,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:49,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:49,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:49,823 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:49,823 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:49,823 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:49,823 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:49,823 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:49,823 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:49,825 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:49,825 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:49,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:49,825 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:49,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:49,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:49,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:49,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:49,826 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:49,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:49,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:49,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:49,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:49,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:49,827 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:49,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:49,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:49,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,830 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,830 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:49,830 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:49,830 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,830 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:49,830 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:49,830 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:49,830 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:49,830 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:49,830 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:49,830 DEBUG [Thread-1666] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:49,830 DEBUG [Thread-1666] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:49,830 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:49,831 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:49,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:49,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:49,831 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:49,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:49,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:49,931 DEBUG [Thread-1666] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:49,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:49,933 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:49,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:49,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:49,934 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:49,934 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:49,934 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:49,934 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:49,934 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:49,934 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:49,935 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:49,935 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:49,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:49,935 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,936 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:49,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:49,936 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:49,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:49,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:49,936 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:49,936 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:49,937 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:49,937 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:49,937 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:49,937 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:49,937 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:49,937 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:49,937 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:49,937 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. started... 2023-03-31 16:54:49,937 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. as already flushing 2023-03-31 16:54:49,937 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:49,937 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:49,937 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:49,937 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,937 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:49,937 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:49,938 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,938 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:49,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:49,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:49,939 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:49,939 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:49,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:49,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:49,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:49,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:49,941 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:49,941 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d5bc773[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:49,941 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:49,941 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:49,942 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:49,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:49,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:49,942 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:49,942 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:49,942 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:49,942 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,942 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:49,944 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:49,944 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,944 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:49,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:49,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:49,944 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:49,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:49,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:49,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:49,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:49,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:49,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:49,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:49,947 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:49,947 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:49,947 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:49,947 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:49,947 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:49,948 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:49,948 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:49,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:49,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:49,948 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:49,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:49,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:49,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:49,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:49,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:49,949 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:49,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:49,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:49,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:49,950 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:49,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:49,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:49,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,958 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:49,958 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:49,958 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:49,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:49,959 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:49,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:49,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:49,959 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:49,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:49,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:49,959 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:49,959 DEBUG [Thread-1666] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:49,959 DEBUG [Thread-1666] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:49,959 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:49,959 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:49,959 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:49,959 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:49,959 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:50,026 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-31 16:54:50,032 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:50,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40768 deadline: 1680281750030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:50,032 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:50,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40794 deadline: 1680281750030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:50,059 DEBUG [Thread-1666] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:50,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:50,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:50,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:50,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:50,065 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:50,065 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:50,065 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:50,065 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:50,065 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:50,065 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:50,067 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:50,067 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:50,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:50,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:50,067 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:50,067 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:50,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:50,067 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:50,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:50,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:50,068 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:50,072 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:50,072 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:50,072 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:50,072 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:50,072 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:50,072 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:50,072 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:50,072 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:50,072 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. started... 2023-03-31 16:54:50,072 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. as already flushing 2023-03-31 16:54:50,072 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:50,072 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:50,072 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:50,072 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:50,072 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:50,072 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:50,074 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:50,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:50,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:50,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:50,074 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:50,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:50,074 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:50,075 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:50,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:50,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:50,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:50,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:50,076 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:50,076 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c39d4c7[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:50,076 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:50,076 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:50,077 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:50,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:50,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:50,077 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:50,078 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:50,078 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:50,078 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:50,078 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:50,080 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:50,080 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:50,080 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:50,080 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:50,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:50,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:50,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:50,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:50,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:50,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:50,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:50,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:50,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:50,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:50,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:50,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:50,083 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:50,083 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:50,083 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:50,083 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:50,083 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:50,085 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:50,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:50,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:50,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:50,085 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:50,085 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:50,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:50,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:50,085 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:50,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:50,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:50,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:50,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:50,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:50,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:50,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:50,087 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:50,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:50,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:50,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:50,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:50,089 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:50,089 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:50,089 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:50,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:50,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:50,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:50,089 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:50,089 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:50,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:50,089 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:50,090 DEBUG [Thread-1666] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:50,089 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:50,090 DEBUG [Thread-1666] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:50,090 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:50,090 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:50,090 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:50,090 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:50,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:50,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:50,151 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=77 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/C/b19e36da9bd44e379cc9225b0a44dbdb 2023-03-31 16:54:50,155 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/A/4a936689ba9243df8d88098f23782fce as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/4a936689ba9243df8d88098f23782fce 2023-03-31 16:54:50,159 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/4a936689ba9243df8d88098f23782fce, entries=150, sequenceid=77, filesize=11.6 K 2023-03-31 16:54:50,160 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/B/4005fb371227416ab321e650f03677de as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/4005fb371227416ab321e650f03677de 2023-03-31 16:54:50,170 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/4005fb371227416ab321e650f03677de, entries=150, sequenceid=77, filesize=11.6 K 2023-03-31 16:54:50,171 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/C/b19e36da9bd44e379cc9225b0a44dbdb as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/b19e36da9bd44e379cc9225b0a44dbdb 2023-03-31 16:54:50,177 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/b19e36da9bd44e379cc9225b0a44dbdb, entries=150, sequenceid=77, filesize=11.6 K 2023-03-31 16:54:50,178 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 3ab680f7c89c77c9be754a4581df5bc8 in 1302ms, sequenceid=77, compaction requested=true 2023-03-31 16:54:50,178 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3ab680f7c89c77c9be754a4581df5bc8: 2023-03-31 16:54:50,178 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:54:50,178 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 16:54:50,178 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 16:54:50,178 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:54:50,179 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:54:50,180 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 16:54:50,180 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1912): 3ab680f7c89c77c9be754a4581df5bc8/A is initiating minor compaction (all files) 2023-03-31 16:54:50,180 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3ab680f7c89c77c9be754a4581df5bc8/A in TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:50,180 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/c682f0805d12414c8ec7285708401af1, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/0c73b5d47d0645778db68c0c3a116615, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/4a936689ba9243df8d88098f23782fce] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp, totalSize=34.8 K 2023-03-31 16:54:50,188 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 16:54:50,188 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): 3ab680f7c89c77c9be754a4581df5bc8/B is initiating minor compaction (all files) 2023-03-31 16:54:50,188 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3ab680f7c89c77c9be754a4581df5bc8/B in TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:50,188 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/892607cbf9fb41b8b3c8688d5135ff94, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/3625c7f038f64ec6a85f794a2e6e0f84, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/4005fb371227416ab321e650f03677de] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp, totalSize=34.8 K 2023-03-31 16:54:50,188 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting c682f0805d12414c8ec7285708401af1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1680281687382 2023-03-31 16:54:50,189 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 0c73b5d47d0645778db68c0c3a116615, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=64, earliestPutTs=1680281687461 2023-03-31 16:54:50,189 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 892607cbf9fb41b8b3c8688d5135ff94, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1680281687382 2023-03-31 16:54:50,189 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 4a936689ba9243df8d88098f23782fce, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=77, earliestPutTs=1680281688563 2023-03-31 16:54:50,189 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 3625c7f038f64ec6a85f794a2e6e0f84, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=64, earliestPutTs=1680281687461 2023-03-31 16:54:50,190 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 4005fb371227416ab321e650f03677de, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=77, earliestPutTs=1680281688563 2023-03-31 16:54:50,191 DEBUG [Thread-1666] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:50,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:50,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:50,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:50,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:50,207 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:50,207 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:50,207 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:50,207 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:50,208 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:50,208 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:50,209 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): 3ab680f7c89c77c9be754a4581df5bc8#B#compaction#215 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:54:50,210 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:50,210 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:50,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:50,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:50,211 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:50,211 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:50,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:50,211 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:50,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:50,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:50,212 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:50,215 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:50,215 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:50,216 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:50,216 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:50,216 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:50,217 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3ab680f7c89c77c9be754a4581df5bc8#A#compaction#216 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:54:50,217 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:50,217 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:50,218 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:50,218 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. started... 2023-03-31 16:54:50,218 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3ab680f7c89c77c9be754a4581df5bc8 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-31 16:54:50,218 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=A 2023-03-31 16:54:50,218 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:50,218 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=B 2023-03-31 16:54:50,218 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:50,218 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=C 2023-03-31 16:54:50,218 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:50,258 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/A/dd5f876a01a84c578e5f6ad2b532086f as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/dd5f876a01a84c578e5f6ad2b532086f 2023-03-31 16:54:50,278 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3ab680f7c89c77c9be754a4581df5bc8/A of 3ab680f7c89c77c9be754a4581df5bc8 into dd5f876a01a84c578e5f6ad2b532086f(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:54:50,278 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3ab680f7c89c77c9be754a4581df5bc8: 2023-03-31 16:54:50,278 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8., storeName=3ab680f7c89c77c9be754a4581df5bc8/A, priority=13, startTime=1680281690178; duration=0sec 2023-03-31 16:54:50,278 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:54:50,278 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 16:54:50,281 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 16:54:50,281 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1912): 3ab680f7c89c77c9be754a4581df5bc8/C is initiating minor compaction (all files) 2023-03-31 16:54:50,281 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3ab680f7c89c77c9be754a4581df5bc8/C in TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:50,281 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/8a23d40f8d7c46a48e3e5f02e537f0c0, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/e5b3f259af94447a8e2f280b1d9ad17e, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/b19e36da9bd44e379cc9225b0a44dbdb] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp, totalSize=34.8 K 2023-03-31 16:54:50,282 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 8a23d40f8d7c46a48e3e5f02e537f0c0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1680281687382 2023-03-31 16:54:50,282 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting e5b3f259af94447a8e2f280b1d9ad17e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=64, earliestPutTs=1680281687461 2023-03-31 16:54:50,283 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting b19e36da9bd44e379cc9225b0a44dbdb, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=77, earliestPutTs=1680281688563 2023-03-31 16:54:50,290 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3ab680f7c89c77c9be754a4581df5bc8#C#compaction#218 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:54:50,635 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/B/71e97fca239244f0a79a062450a36929 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/71e97fca239244f0a79a062450a36929 2023-03-31 16:54:50,640 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3ab680f7c89c77c9be754a4581df5bc8/B of 3ab680f7c89c77c9be754a4581df5bc8 into 71e97fca239244f0a79a062450a36929(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:54:50,640 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3ab680f7c89c77c9be754a4581df5bc8: 2023-03-31 16:54:50,640 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8., storeName=3ab680f7c89c77c9be754a4581df5bc8/B, priority=13, startTime=1680281690178; duration=0sec 2023-03-31 16:54:50,640 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:54:50,655 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=96 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/A/971dc5a195be4ed8a2f7cb274db0afae 2023-03-31 16:54:50,673 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=96 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/B/dd7de93a1c9844dfbd827fd4348d72b0 2023-03-31 16:54:50,685 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=96 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/C/3d88d8c9451f4f5cad60aff900879f35 2023-03-31 16:54:50,697 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/A/971dc5a195be4ed8a2f7cb274db0afae as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/971dc5a195be4ed8a2f7cb274db0afae 2023-03-31 16:54:50,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on 3ab680f7c89c77c9be754a4581df5bc8 2023-03-31 16:54:50,701 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. as already flushing 2023-03-31 16:54:50,702 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/971dc5a195be4ed8a2f7cb274db0afae, entries=150, sequenceid=96, filesize=11.6 K 2023-03-31 16:54:50,705 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/B/dd7de93a1c9844dfbd827fd4348d72b0 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/dd7de93a1c9844dfbd827fd4348d72b0 2023-03-31 16:54:50,710 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/C/8f9289fe2d404586b88faedd39c55668 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/8f9289fe2d404586b88faedd39c55668 2023-03-31 16:54:50,713 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/dd7de93a1c9844dfbd827fd4348d72b0, entries=150, sequenceid=96, filesize=11.6 K 2023-03-31 16:54:50,714 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/C/3d88d8c9451f4f5cad60aff900879f35 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/3d88d8c9451f4f5cad60aff900879f35 2023-03-31 16:54:50,717 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3ab680f7c89c77c9be754a4581df5bc8/C of 3ab680f7c89c77c9be754a4581df5bc8 into 8f9289fe2d404586b88faedd39c55668(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:54:50,717 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3ab680f7c89c77c9be754a4581df5bc8: 2023-03-31 16:54:50,717 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8., storeName=3ab680f7c89c77c9be754a4581df5bc8/C, priority=13, startTime=1680281690179; duration=0sec 2023-03-31 16:54:50,717 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:54:50,722 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/3d88d8c9451f4f5cad60aff900879f35, entries=150, sequenceid=96, filesize=11.6 K 2023-03-31 16:54:50,723 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=33.54 KB/34350 for 3ab680f7c89c77c9be754a4581df5bc8 in 505ms, sequenceid=96, compaction requested=false 2023-03-31 16:54:50,723 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3ab680f7c89c77c9be754a4581df5bc8: 2023-03-31 16:54:50,723 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:50,724 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:50,724 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:50,724 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:50,724 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:50,724 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:50,725 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:50,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:50,725 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:50,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:50,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:50,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:50,726 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:50,726 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:50,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:50,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:50,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:50,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:50,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on 3ab680f7c89c77c9be754a4581df5bc8 2023-03-31 16:54:50,727 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3ab680f7c89c77c9be754a4581df5bc8 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-31 16:54:50,727 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:50,727 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=A 2023-03-31 16:54:50,728 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:50,728 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:50,728 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10ec188b[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:50,728 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:50,728 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=B 2023-03-31 16:54:50,728 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:50,728 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=C 2023-03-31 16:54:50,728 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:50,729 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:50,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:50,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:50,729 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:50,729 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:50,729 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:50,729 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:50,729 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:50,731 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:50,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:50,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:50,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:50,731 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:50,731 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:50,731 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:50,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:50,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:50,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:50,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:50,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:50,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:50,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:50,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:50,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:50,734 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:50,734 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:50,734 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:50,734 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:50,735 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:50,737 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:50,737 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:50,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:50,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:50,737 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:50,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:50,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:50,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:50,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:50,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:50,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:50,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:50,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:50,738 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:50,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:50,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:50,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:50,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:50,740 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:50,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:50,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:50,742 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:50,742 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:50,742 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:50,742 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:50,743 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:50,743 DEBUG [Thread-1666] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:50,743 DEBUG [Thread-1666] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:50,742 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:50,743 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:50,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:50,742 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:50,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:50,743 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:50,743 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:50,743 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:50,743 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:50,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:50,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:50,784 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680281690773, max timestamp=1680281690782Num uniques 859; 2023-03-31 16:54:50,784 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680281690773, max timestamp=1680281690782Num uniques 859; 2023-03-31 16:54:50,784 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 16:54:50,784 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680281690773, max timestamp=1680281690782Num uniques 859; 2023-03-31 16:54:50,784 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 16:54:50,784 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 16:54:50,839 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:50,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40748 deadline: 1680281750837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:50,843 DEBUG [Thread-1666] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:50,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:50,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:50,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:50,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:50,846 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:50,846 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:50,846 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:50,846 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:50,846 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:50,846 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:50,847 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:50,847 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:50,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:50,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:50,848 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:50,848 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:50,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:50,848 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:50,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:50,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:50,848 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:50,849 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:50,849 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:50,849 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:50,849 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:50,849 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:50,849 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:50,849 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:50,849 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:50,849 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. started... 2023-03-31 16:54:50,849 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. as already flushing 2023-03-31 16:54:50,849 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:50,849 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:50,849 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:50,849 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:50,850 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:50,850 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:50,851 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:50,851 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:50,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:50,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:50,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:50,851 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:50,851 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:50,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:50,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:50,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:50,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:50,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:50,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:50,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@367ad69f[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:50,852 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:50,853 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:50,855 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:50,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:50,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:50,855 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:50,855 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:50,855 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:50,855 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:50,855 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:50,856 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:50,856 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:50,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:50,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:50,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:50,856 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:50,856 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:50,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:50,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:50,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:50,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:50,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:50,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:50,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:50,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:50,859 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:50,859 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:50,859 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:50,859 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:50,859 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:50,859 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:50,860 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:50,860 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:50,861 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:50,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:50,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:50,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:50,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:50,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:50,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:50,861 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:50,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:50,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:50,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:50,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:50,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:50,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:50,862 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:50,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:50,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:50,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:50,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:50,864 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:50,864 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:50,864 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:50,864 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:50,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:50,864 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:50,864 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:50,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:50,864 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:50,865 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:50,865 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:50,865 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:50,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:50,865 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:50,865 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:50,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:50,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:50,865 DEBUG [Thread-1666] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:50,865 DEBUG [Thread-1666] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:50,941 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:50,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40748 deadline: 1680281750940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:50,965 DEBUG [Thread-1666] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:50,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:50,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:50,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:50,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:50,969 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:50,969 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:50,969 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:50,969 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:50,969 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:50,969 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:50,974 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:50,974 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:50,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:50,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:50,975 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:50,975 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:50,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:50,975 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:50,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:50,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:50,975 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:50,976 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:50,976 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:50,976 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:50,976 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:50,976 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:50,977 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:50,977 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:50,977 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:50,977 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. started... 2023-03-31 16:54:50,977 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. as already flushing 2023-03-31 16:54:50,977 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:50,977 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:50,977 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:50,977 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:50,977 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:50,977 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:50,981 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:50,981 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:50,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:50,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:50,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:50,982 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:50,982 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:50,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:50,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:50,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:50,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:50,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:50,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:50,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2509ef7c[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:50,983 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:50,983 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:50,984 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:50,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:50,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:50,984 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:50,984 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:50,984 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:50,985 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:50,984 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:50,986 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:50,986 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:50,986 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:50,986 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:50,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:50,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:50,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:50,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:50,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:50,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:50,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:50,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:50,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:50,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:50,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:50,988 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:50,988 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:50,988 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:50,988 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:50,988 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:50,988 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:50,991 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:50,991 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:50,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:50,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:50,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:50,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:50,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:50,991 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:50,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:50,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:50,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:50,992 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:50,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:50,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:50,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:50,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:50,993 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:50,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:50,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:50,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:50,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:50,995 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:50,995 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:50,996 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:50,996 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:50,996 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:50,996 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:50,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:50,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:50,996 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:50,996 DEBUG [Thread-1666] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:50,996 DEBUG [Thread-1666] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:50,996 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:50,996 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:50,996 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:50,996 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:50,996 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:50,996 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:50,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:50,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:51,034 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:51,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40768 deadline: 1680281751033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,039 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:51,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40794 deadline: 1680281751037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,096 DEBUG [Thread-1666] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:51,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:51,098 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:51,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:51,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:51,099 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:51,099 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:51,099 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:51,099 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:51,100 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:51,100 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:51,102 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:51,102 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:51,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:51,103 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,103 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:51,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:51,103 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:51,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:51,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:51,103 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:51,104 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:51,104 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:51,104 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:51,104 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:51,104 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:51,104 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:51,104 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:51,104 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:51,104 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. started... 2023-03-31 16:54:51,105 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. as already flushing 2023-03-31 16:54:51,105 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:51,105 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:51,105 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:51,105 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,105 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:51,105 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:51,106 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,106 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:51,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:51,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:51,107 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:51,107 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:51,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:51,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:51,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:51,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:51,108 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:51,108 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@378498f6[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:51,108 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:51,108 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:51,109 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:51,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:51,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:51,109 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:51,110 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:51,110 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:51,110 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,110 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:51,111 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,111 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:51,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:51,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:51,111 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:51,111 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:51,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:51,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:51,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:51,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:51,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:51,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:51,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:51,113 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:51,113 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:51,113 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:51,113 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:51,113 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:51,115 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:51,115 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:51,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:51,116 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:51,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:51,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:51,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:51,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:51,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:51,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:51,116 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:51,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:51,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:51,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:51,117 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:51,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:51,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:51,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,119 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,119 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:51,119 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:51,119 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:51,119 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:51,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:51,120 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:51,120 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:51,120 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:51,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:51,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:51,120 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:51,120 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:51,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:51,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:51,120 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:51,120 DEBUG [Thread-1666] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:51,120 DEBUG [Thread-1666] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:51,146 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:51,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40748 deadline: 1680281751144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,150 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=109 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/A/dfc18ad9987c40bebca50113a82365fd 2023-03-31 16:54:51,160 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=109 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/B/bed3473d9b2046fb94a33c30ec707294 2023-03-31 16:54:51,220 DEBUG [Thread-1666] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:51,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:51,223 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:51,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:51,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:51,224 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:51,224 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:51,224 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:51,224 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:51,225 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:51,225 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:51,226 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,226 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:51,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:51,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:51,226 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,226 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:51,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:51,227 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:51,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:51,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:51,227 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:51,227 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:51,227 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:51,227 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:51,227 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:51,227 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:51,228 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:51,228 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:51,228 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:51,228 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. started... 2023-03-31 16:54:51,228 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. as already flushing 2023-03-31 16:54:51,228 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:51,228 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:51,228 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:51,228 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,228 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:51,228 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:51,229 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,229 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:51,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:51,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:51,230 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:51,230 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:51,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:51,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:51,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:51,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:51,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:51,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47dcf6cd[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:51,231 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:51,231 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:51,233 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:51,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:51,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:51,233 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:51,233 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:51,233 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:51,234 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,234 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:51,235 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:51,236 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:51,236 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:51,236 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:51,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:51,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:51,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:51,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:51,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:51,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:51,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:51,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:51,252 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:51,252 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:51,252 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:51,252 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:51,253 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:51,254 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:51,254 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:51,254 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:51,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:51,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:51,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:51,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:51,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:51,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:51,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:51,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:51,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:51,255 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:51,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:51,256 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:51,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:51,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:51,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,259 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,259 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:51,259 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:51,259 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,259 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:51,259 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:51,259 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:51,259 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:51,259 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:51,259 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:51,259 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:51,259 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:51,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:51,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:51,259 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:51,259 DEBUG [Thread-1666] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:51,259 DEBUG [Thread-1666] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:51,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:51,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:51,359 DEBUG [Thread-1666] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:51,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:51,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:51,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:51,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:51,363 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:51,363 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:51,363 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:51,363 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:51,363 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:51,363 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:51,365 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,365 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:51,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:51,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:51,365 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,365 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:51,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:51,366 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:51,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:51,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:51,366 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:51,367 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:51,367 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:51,367 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:51,367 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:51,367 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:51,367 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:51,367 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:51,367 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:51,367 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. started... 2023-03-31 16:54:51,367 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. as already flushing 2023-03-31 16:54:51,367 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:51,367 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:51,367 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:51,367 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,367 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:51,367 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:51,369 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,369 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:51,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:51,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:51,369 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:51,369 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:51,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:51,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:51,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:51,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:51,370 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:51,370 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27f21e87[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:51,370 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:51,371 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:51,371 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:51,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:51,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:51,372 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:51,372 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:51,372 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:51,372 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,372 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:51,374 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,374 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:51,374 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:51,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:51,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:51,374 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:51,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:51,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:51,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:51,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:51,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:51,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:51,376 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:51,377 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:51,377 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:51,377 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:51,377 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:51,377 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:51,378 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:51,378 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:51,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:51,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:51,378 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:51,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:51,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:51,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:51,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:51,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:51,379 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:51,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:51,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:51,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:51,380 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:51,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:51,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:51,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,382 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,382 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:51,383 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:51,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:51,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:51,383 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:51,383 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:51,383 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,383 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:51,383 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:51,383 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:51,383 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:51,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:51,383 DEBUG [Thread-1666] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:51,383 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:51,383 DEBUG [Thread-1666] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:51,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:51,383 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:51,383 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:51,463 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:51,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40748 deadline: 1680281751460, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,483 DEBUG [Thread-1666] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:51,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:51,485 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:51,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:51,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:51,486 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:51,486 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:51,486 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:51,486 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:51,487 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:51,487 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:51,489 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:51,489 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:51,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:51,489 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,489 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:51,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:51,490 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:51,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:51,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:51,490 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:51,490 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:51,490 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:51,490 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:51,491 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:51,491 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:51,491 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:51,491 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:51,491 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:51,491 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. started... 2023-03-31 16:54:51,491 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. as already flushing 2023-03-31 16:54:51,491 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:51,491 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:51,491 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:51,491 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,491 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:51,491 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:51,493 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,493 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:51,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:51,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:51,493 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:51,493 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:51,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:51,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:51,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:51,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:51,495 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:51,495 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4dea770a[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:51,495 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:51,496 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:51,496 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:51,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:51,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:51,497 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:51,497 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:51,497 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:51,497 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,497 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:51,498 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,498 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:51,498 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:51,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:51,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:51,498 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:51,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:51,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:51,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:51,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:51,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:51,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:51,501 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:51,501 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:51,501 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:51,501 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:51,501 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:51,501 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:51,504 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:51,504 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:51,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:51,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:51,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:51,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:51,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:51,504 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:51,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:51,504 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:51,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:51,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:51,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:51,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:51,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:51,505 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:51,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:51,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,507 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:51,508 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,508 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:51,508 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:51,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:51,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:51,508 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:51,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:51,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:51,508 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:51,508 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,508 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:51,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:51,508 DEBUG [Thread-1666] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:51,508 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:51,508 DEBUG [Thread-1666] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:51,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:51,508 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:51,570 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=109 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/C/305c2b13981440039c54d0b08b23188b 2023-03-31 16:54:51,574 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/A/dfc18ad9987c40bebca50113a82365fd as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/dfc18ad9987c40bebca50113a82365fd 2023-03-31 16:54:51,578 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/dfc18ad9987c40bebca50113a82365fd, entries=200, sequenceid=109, filesize=13.9 K 2023-03-31 16:54:51,579 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/B/bed3473d9b2046fb94a33c30ec707294 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/bed3473d9b2046fb94a33c30ec707294 2023-03-31 16:54:51,582 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/bed3473d9b2046fb94a33c30ec707294, entries=150, sequenceid=109, filesize=11.6 K 2023-03-31 16:54:51,585 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/C/305c2b13981440039c54d0b08b23188b as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/305c2b13981440039c54d0b08b23188b 2023-03-31 16:54:51,589 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/305c2b13981440039c54d0b08b23188b, entries=150, sequenceid=109, filesize=11.6 K 2023-03-31 16:54:51,590 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 3ab680f7c89c77c9be754a4581df5bc8 in 863ms, sequenceid=109, compaction requested=true 2023-03-31 16:54:51,590 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3ab680f7c89c77c9be754a4581df5bc8: 2023-03-31 16:54:51,590 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:54:51,590 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 16:54:51,591 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 16:54:51,590 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:54:51,591 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:54:51,592 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38101 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 16:54:51,592 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 16:54:51,592 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1912): 3ab680f7c89c77c9be754a4581df5bc8/A is initiating minor compaction (all files) 2023-03-31 16:54:51,593 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): 3ab680f7c89c77c9be754a4581df5bc8/B is initiating minor compaction (all files) 2023-03-31 16:54:51,593 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3ab680f7c89c77c9be754a4581df5bc8/A in TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:51,593 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3ab680f7c89c77c9be754a4581df5bc8/B in TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:51,593 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/dd5f876a01a84c578e5f6ad2b532086f, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/971dc5a195be4ed8a2f7cb274db0afae, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/dfc18ad9987c40bebca50113a82365fd] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp, totalSize=37.2 K 2023-03-31 16:54:51,593 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/71e97fca239244f0a79a062450a36929, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/dd7de93a1c9844dfbd827fd4348d72b0, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/bed3473d9b2046fb94a33c30ec707294] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp, totalSize=34.9 K 2023-03-31 16:54:51,594 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting dd5f876a01a84c578e5f6ad2b532086f, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=77, earliestPutTs=1680281688563 2023-03-31 16:54:51,594 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 71e97fca239244f0a79a062450a36929, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=77, earliestPutTs=1680281688563 2023-03-31 16:54:51,594 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 971dc5a195be4ed8a2f7cb274db0afae, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1680281688909 2023-03-31 16:54:51,594 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting dd7de93a1c9844dfbd827fd4348d72b0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1680281688909 2023-03-31 16:54:51,594 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting dfc18ad9987c40bebca50113a82365fd, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=109, earliestPutTs=1680281690713 2023-03-31 16:54:51,594 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting bed3473d9b2046fb94a33c30ec707294, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=109, earliestPutTs=1680281690713 2023-03-31 16:54:51,600 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3ab680f7c89c77c9be754a4581df5bc8#A#compaction#224 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:54:51,602 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): 3ab680f7c89c77c9be754a4581df5bc8#B#compaction#225 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:54:51,608 DEBUG [Thread-1666] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:51,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on 3ab680f7c89c77c9be754a4581df5bc8 2023-03-31 16:54:51,609 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3ab680f7c89c77c9be754a4581df5bc8 3/3 column families, dataSize=140.89 KB heapSize=405.61 KB 2023-03-31 16:54:51,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:51,609 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=A 2023-03-31 16:54:51,609 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-31 16:54:51,609 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=B 2023-03-31 16:54:51,609 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-31 16:54:51,609 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=C 2023-03-31 16:54:51,609 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-31 16:54:51,618 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:51,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:51,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:51,621 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:51,622 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:51,622 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:51,622 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:51,624 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:51,624 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:51,624 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:51,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40716 deadline: 1680281751618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,624 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:51,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40750 deadline: 1680281751619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,625 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,626 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,626 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:51,627 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:51,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:51,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:51,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:51,628 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:51,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:51,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:51,628 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:51,630 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:51,630 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:51,630 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:51,630 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:51,630 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:51,630 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:51,630 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:51,630 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:51,630 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. started... 2023-03-31 16:54:51,630 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/B/e997c7a82ddb4f5ea3121e9784db51ab as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/e997c7a82ddb4f5ea3121e9784db51ab 2023-03-31 16:54:51,631 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. as already flushing 2023-03-31 16:54:51,631 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:51,631 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:51,631 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:51,631 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,631 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:51,631 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:51,633 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:51,633 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:51,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:51,633 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:51,633 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:51,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:51,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:51,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:51,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,635 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3ab680f7c89c77c9be754a4581df5bc8/B of 3ab680f7c89c77c9be754a4581df5bc8 into e997c7a82ddb4f5ea3121e9784db51ab(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:54:51,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:51,635 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3ab680f7c89c77c9be754a4581df5bc8: 2023-03-31 16:54:51,635 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8., storeName=3ab680f7c89c77c9be754a4581df5bc8/B, priority=13, startTime=1680281691590; duration=0sec 2023-03-31 16:54:51,635 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:54:51,635 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:51,635 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 16:54:51,635 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:51,635 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5eba0569[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:51,635 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:51,636 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 16:54:51,636 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): 3ab680f7c89c77c9be754a4581df5bc8/C is initiating minor compaction (all files) 2023-03-31 16:54:51,636 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3ab680f7c89c77c9be754a4581df5bc8/C in TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:51,636 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/8f9289fe2d404586b88faedd39c55668, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/3d88d8c9451f4f5cad60aff900879f35, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/305c2b13981440039c54d0b08b23188b] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp, totalSize=34.9 K 2023-03-31 16:54:51,637 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 8f9289fe2d404586b88faedd39c55668, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=77, earliestPutTs=1680281688563 2023-03-31 16:54:51,637 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 3d88d8c9451f4f5cad60aff900879f35, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1680281688909 2023-03-31 16:54:51,638 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 305c2b13981440039c54d0b08b23188b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=109, earliestPutTs=1680281690713 2023-03-31 16:54:51,640 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:51,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:51,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:51,641 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:51,641 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:51,641 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:51,641 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,641 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:51,642 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:51,642 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:51,642 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,642 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:51,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:51,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:51,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:51,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:51,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:51,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,643 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): 3ab680f7c89c77c9be754a4581df5bc8#C#compaction#227 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:54:51,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:51,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:51,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:51,645 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:51,645 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:51,645 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:51,645 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:51,645 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:51,645 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:51,646 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:51,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:51,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:51,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:51,646 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:51,646 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:51,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:51,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:51,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:51,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:51,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:51,647 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:51,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:51,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:51,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:51,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,649 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:51,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:51,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,651 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:51,651 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:51,651 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,651 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:51,651 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:51,651 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:51,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:51,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:51,651 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,652 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:51,652 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:51,652 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:51,651 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:51,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:51,652 DEBUG [Thread-1666] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:51,652 DEBUG [Thread-1666] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:51,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:51,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:51,727 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:51,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40750 deadline: 1680281751725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,731 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:51,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40716 deadline: 1680281751727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,752 DEBUG [Thread-1666] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:51,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:51,754 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:51,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:51,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:51,755 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:51,755 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:51,755 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:51,755 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:51,756 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:51,756 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:51,758 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,759 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:51,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:51,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:51,759 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,759 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:51,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:51,759 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:51,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:51,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:51,760 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:51,760 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:51,760 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:51,760 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:51,760 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:51,760 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:51,760 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:51,760 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:51,760 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:51,760 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. started... 2023-03-31 16:54:51,760 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. as already flushing 2023-03-31 16:54:51,760 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:51,760 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:51,761 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:51,761 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,761 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:51,761 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:51,762 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:51,762 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:51,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:51,762 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:51,762 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:51,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:51,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:51,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:51,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:51,764 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:51,764 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e70eaed[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:51,764 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:51,764 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:51,765 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:51,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:51,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:51,765 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:51,765 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:51,765 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:51,765 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,765 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:51,770 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:51,771 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:51,770 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,771 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:51,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:51,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:51,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:51,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:51,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:51,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:51,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:51,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:51,773 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:51,773 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:51,773 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:51,773 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:51,773 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:51,773 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:51,775 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:51,775 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:51,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:51,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:51,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:51,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:51,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:51,775 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:51,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:51,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:51,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:51,775 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:51,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:51,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:51,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:51,776 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:51,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:51,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,779 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,779 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:51,779 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:51,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:51,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:51,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:51,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:51,779 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:51,779 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:51,779 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:51,779 DEBUG [Thread-1666] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:51,779 DEBUG [Thread-1666] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:51,779 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,779 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:51,779 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:51,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:51,780 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:51,779 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:51,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:51,879 DEBUG [Thread-1666] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:51,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:51,882 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:51,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:51,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:51,883 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:51,883 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:51,883 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:51,883 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:51,884 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:51,884 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:51,885 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:51,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:51,885 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:51,885 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,885 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:51,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:51,886 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:51,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:51,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:51,886 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:51,886 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:51,886 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:51,886 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:51,887 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:51,887 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:51,887 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:51,887 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:51,887 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:51,887 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. started... 2023-03-31 16:54:51,887 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. as already flushing 2023-03-31 16:54:51,887 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:51,887 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:51,887 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:51,887 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,887 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:51,887 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:51,890 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,890 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:51,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:51,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:51,890 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:51,890 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:51,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:51,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:51,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:51,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:51,892 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:51,892 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:51,892 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4941c582[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:51,892 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:51,893 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:51,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:51,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:51,893 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:51,893 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:51,893 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:51,893 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,893 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:51,894 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:51,894 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:51,894 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,894 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:51,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:51,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:51,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:51,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:51,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:51,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:51,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:51,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:51,896 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:51,896 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:51,897 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:51,897 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:51,897 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:51,897 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:51,898 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:51,898 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:51,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:51,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:51,899 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:51,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:51,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:51,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:51,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:51,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:51,899 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:51,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:51,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:51,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:51,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:51,900 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:51,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:51,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,904 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,904 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:51,904 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:51,904 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:51,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:51,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:51,904 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:51,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:51,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:51,904 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:51,904 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,904 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:51,904 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:51,904 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:51,904 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:51,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:51,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:51,905 DEBUG [Thread-1666] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:51,905 DEBUG [Thread-1666] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:51,929 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:51,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40750 deadline: 1680281751928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,934 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:51,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40716 deadline: 1680281751933, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:51,965 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:51,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40748 deadline: 1680281751964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:52,005 DEBUG [Thread-1666] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:52,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:52,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:52,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:52,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:52,008 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:52,008 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:52,008 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:52,008 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:52,009 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:52,009 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:52,010 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:52,010 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:52,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:52,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:52,010 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:52,010 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:52,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:52,010 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:52,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:52,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:52,011 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:52,011 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:52,011 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:52,011 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:52,011 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:52,011 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:52,012 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:52,012 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:52,012 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:52,012 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. started... 2023-03-31 16:54:52,012 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. as already flushing 2023-03-31 16:54:52,012 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:52,012 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:52,012 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:52,012 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:52,012 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:52,012 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:52,014 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:52,014 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:52,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:52,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:52,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:52,014 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:52,014 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:52,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:52,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:52,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:52,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:52,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:52,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:52,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@578db987[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:52,016 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:52,016 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:52,017 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:52,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:52,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:52,017 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:52,017 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:52,017 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:52,017 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:52,017 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:52,019 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/A/6e2435bd4c00454db912d5779bcdb3de as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/6e2435bd4c00454db912d5779bcdb3de 2023-03-31 16:54:52,019 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:52,019 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:52,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:52,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:52,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:52,019 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:52,020 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:52,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:52,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:52,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:52,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:52,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:52,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:52,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:52,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:52,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:52,022 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:52,022 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:52,022 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:52,022 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:52,022 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:52,023 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3ab680f7c89c77c9be754a4581df5bc8/A of 3ab680f7c89c77c9be754a4581df5bc8 into 6e2435bd4c00454db912d5779bcdb3de(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:54:52,024 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3ab680f7c89c77c9be754a4581df5bc8: 2023-03-31 16:54:52,024 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8., storeName=3ab680f7c89c77c9be754a4581df5bc8/A, priority=13, startTime=1680281691590; duration=0sec 2023-03-31 16:54:52,024 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:54:52,028 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=147 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/A/635d2c14218f4906bfe16bb619d60aa8 2023-03-31 16:54:52,028 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:52,029 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:52,030 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:52,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:52,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:52,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:52,031 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:52,031 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:52,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:52,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:52,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:52,032 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:52,032 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:52,032 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:52,032 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:52,032 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:52,032 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:52,033 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:52,032 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:52,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:52,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:52,033 DEBUG [Thread-1666] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:52,033 DEBUG [Thread-1666] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:52,033 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:52,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:52,033 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:52,033 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:52,033 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:52,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:52,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:52,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:52,034 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:52,062 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/C/fa817bcbf06040269857bfc06c6f8a62 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/fa817bcbf06040269857bfc06c6f8a62 2023-03-31 16:54:52,067 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3ab680f7c89c77c9be754a4581df5bc8/C of 3ab680f7c89c77c9be754a4581df5bc8 into fa817bcbf06040269857bfc06c6f8a62(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:54:52,067 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3ab680f7c89c77c9be754a4581df5bc8: 2023-03-31 16:54:52,067 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8., storeName=3ab680f7c89c77c9be754a4581df5bc8/C, priority=13, startTime=1680281691591; duration=0sec 2023-03-31 16:54:52,067 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:54:52,083 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=147 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/B/9717f3a6d39e4fc1975721d3169d0ca3 2023-03-31 16:54:52,110 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=147 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/C/5302cd4baa30418996ac4ea47c688c8e 2023-03-31 16:54:52,115 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/A/635d2c14218f4906bfe16bb619d60aa8 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/635d2c14218f4906bfe16bb619d60aa8 2023-03-31 16:54:52,119 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/635d2c14218f4906bfe16bb619d60aa8, entries=150, sequenceid=147, filesize=11.8 K 2023-03-31 16:54:52,120 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/B/9717f3a6d39e4fc1975721d3169d0ca3 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/9717f3a6d39e4fc1975721d3169d0ca3 2023-03-31 16:54:52,126 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/9717f3a6d39e4fc1975721d3169d0ca3, entries=150, sequenceid=147, filesize=11.8 K 2023-03-31 16:54:52,127 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/C/5302cd4baa30418996ac4ea47c688c8e as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/5302cd4baa30418996ac4ea47c688c8e 2023-03-31 16:54:52,130 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/5302cd4baa30418996ac4ea47c688c8e, entries=150, sequenceid=147, filesize=11.8 K 2023-03-31 16:54:52,131 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~405.56 KB/415296, currentSize=40.25 KB/41220 for 3ab680f7c89c77c9be754a4581df5bc8 in 522ms, sequenceid=147, compaction requested=false 2023-03-31 16:54:52,131 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3ab680f7c89c77c9be754a4581df5bc8: 2023-03-31 16:54:52,133 DEBUG [Thread-1666] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:52,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:52,136 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:52,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:52,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:52,138 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:52,138 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:52,138 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:52,138 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:52,139 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:52,139 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:52,140 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:52,140 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:52,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:52,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:52,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:52,141 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:52,141 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:52,141 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:52,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:52,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:52,141 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:52,142 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:52,142 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:52,142 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:52,142 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:52,142 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:52,144 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:52,144 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:52,144 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:52,144 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. started... 2023-03-31 16:54:52,144 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3ab680f7c89c77c9be754a4581df5bc8 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-03-31 16:54:52,144 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=A 2023-03-31 16:54:52,145 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:52,145 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=B 2023-03-31 16:54:52,145 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:52,145 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=C 2023-03-31 16:54:52,145 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:52,158 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=159 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/A/e35bb37161cb489ba3eea7d527b760fe 2023-03-31 16:54:52,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on 3ab680f7c89c77c9be754a4581df5bc8 2023-03-31 16:54:52,232 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. as already flushing 2023-03-31 16:54:52,258 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680281692248, max timestamp=1680281692256Num uniques 859; 2023-03-31 16:54:52,258 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 16:54:52,258 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680281692248, max timestamp=1680281692256Num uniques 859; 2023-03-31 16:54:52,258 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 16:54:52,258 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680281692248, max timestamp=1680281692256Num uniques 859; 2023-03-31 16:54:52,258 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 16:54:52,291 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680281692270, max timestamp=1680281692288Num uniques 1018; 2023-03-31 16:54:52,291 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-03-31 16:54:52,291 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680281692270, max timestamp=1680281692288Num uniques 1018; 2023-03-31 16:54:52,292 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-03-31 16:54:52,293 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7682, totalHeapSize=17712, min timestamp=1680281692270, max timestamp=1680281692288Num uniques 1018; 2023-03-31 16:54:52,293 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46612, new segment data size=7682, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-03-31 16:54:52,329 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680281692317, max timestamp=1680281692326Num uniques 1027; 2023-03-31 16:54:52,329 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-03-31 16:54:52,330 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680281692317, max timestamp=1680281692326Num uniques 1027; 2023-03-31 16:54:52,330 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680281692317, max timestamp=1680281692326Num uniques 1027; 2023-03-31 16:54:52,330 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-03-31 16:54:52,330 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-03-31 16:54:52,367 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:52,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40750 deadline: 1680281752367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:52,368 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:52,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40716 deadline: 1680281752367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:52,469 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:52,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40750 deadline: 1680281752468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:52,472 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:52,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40716 deadline: 1680281752470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:52,571 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=159 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/B/bd9851dfc85a48a7b39c35ae34a368a1 2023-03-31 16:54:52,585 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=159 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/C/48ee8e530bf848169a416a1ad8cb40dc 2023-03-31 16:54:52,591 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/A/e35bb37161cb489ba3eea7d527b760fe as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/e35bb37161cb489ba3eea7d527b760fe 2023-03-31 16:54:52,595 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/e35bb37161cb489ba3eea7d527b760fe, entries=100, sequenceid=159, filesize=9.4 K 2023-03-31 16:54:52,595 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/B/bd9851dfc85a48a7b39c35ae34a368a1 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/bd9851dfc85a48a7b39c35ae34a368a1 2023-03-31 16:54:52,601 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/bd9851dfc85a48a7b39c35ae34a368a1, entries=100, sequenceid=159, filesize=9.4 K 2023-03-31 16:54:52,601 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/C/48ee8e530bf848169a416a1ad8cb40dc as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/48ee8e530bf848169a416a1ad8cb40dc 2023-03-31 16:54:52,605 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/48ee8e530bf848169a416a1ad8cb40dc, entries=100, sequenceid=159, filesize=9.4 K 2023-03-31 16:54:52,605 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 3ab680f7c89c77c9be754a4581df5bc8 in 461ms, sequenceid=159, compaction requested=true 2023-03-31 16:54:52,605 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3ab680f7c89c77c9be754a4581df5bc8: 2023-03-31 16:54:52,605 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:52,605 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:52,605 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:52,605 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:52,606 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:52,606 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:52,607 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:52,607 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:52,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:52,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:52,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:52,607 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:52,607 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:52,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:52,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:52,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:52,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:52,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:52,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:52,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59127f52[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:52,609 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:52,609 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:52,610 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:52,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:52,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:52,611 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:52,611 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:52,611 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:52,611 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:52,611 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:52,612 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:52,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:52,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:52,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:52,612 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:52,612 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:52,612 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:52,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:52,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:52,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:52,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:52,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:52,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:52,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:52,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:52,616 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:52,616 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:52,616 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:52,616 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:52,616 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:52,616 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:52,619 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:52,619 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:52,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:52,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:52,619 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:52,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:52,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:52,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:52,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:52,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:52,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:52,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:52,620 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:52,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:52,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:52,622 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:52,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:52,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:52,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:52,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:52,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:52,627 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:52,627 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:52,627 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:52,627 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:52,627 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:52,627 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:52,627 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:52,627 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:52,627 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:52,627 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:52,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:52,627 DEBUG [Thread-1666] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:52,627 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:52,627 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:52,627 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:52,628 DEBUG [Thread-1666] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:52,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:52,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:52,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:52,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on 3ab680f7c89c77c9be754a4581df5bc8 2023-03-31 16:54:52,674 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680281692355, max timestamp=1680281692671Num uniques 1086; 2023-03-31 16:54:52,674 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-03-31 16:54:52,675 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680281692355, max timestamp=1680281692671Num uniques 1086; 2023-03-31 16:54:52,675 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-03-31 16:54:52,675 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680281692355, max timestamp=1680281692671Num uniques 1086; 2023-03-31 16:54:52,675 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-03-31 16:54:52,675 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3ab680f7c89c77c9be754a4581df5bc8 3/3 column families, dataSize=26.84 KB heapSize=63.70 KB 2023-03-31 16:54:52,675 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=A 2023-03-31 16:54:52,675 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:54:52,675 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=B 2023-03-31 16:54:52,675 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:54:52,675 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=C 2023-03-31 16:54:52,675 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:54:52,684 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=231 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/A/613a872127fb4c298ce182599b1d6a61 2023-03-31 16:54:52,709 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680281692690, max timestamp=1680281692708Num uniques 859; 2023-03-31 16:54:52,709 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 16:54:52,710 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680281692690, max timestamp=1680281692708Num uniques 859; 2023-03-31 16:54:52,710 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 16:54:52,710 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680281692690, max timestamp=1680281692708Num uniques 859; 2023-03-31 16:54:52,710 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 16:54:52,728 DEBUG [Thread-1666] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:52,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:52,730 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:52,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:52,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:52,731 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:52,731 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:52,731 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:52,731 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:52,731 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:52,731 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:52,732 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:52,732 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:52,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:52,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:52,733 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:52,733 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:52,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:52,733 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:52,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:52,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:52,733 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:52,733 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:52,734 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:52,734 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:52,734 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:52,734 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:52,734 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:52,734 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:52,734 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:52,734 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. started... 2023-03-31 16:54:52,734 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. as already flushing 2023-03-31 16:54:52,734 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:52,734 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:52,734 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:52,734 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:52,734 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:52,734 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:52,736 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:52,736 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:52,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:52,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:52,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:52,736 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:52,737 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:52,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:52,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:52,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:52,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:52,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:52,738 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:52,738 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20ccc11f[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:52,738 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:52,738 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:52,739 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:52,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:52,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:52,739 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:52,739 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:52,739 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:52,739 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:52,739 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:52,741 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:52,741 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:52,741 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:52,741 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:52,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:52,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:52,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:52,742 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680281692726, max timestamp=1680281692740Num uniques 1018; 2023-03-31 16:54:52,742 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-03-31 16:54:52,743 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680281692726, max timestamp=1680281692740Num uniques 1018; 2023-03-31 16:54:52,743 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-03-31 16:54:52,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:52,743 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680281692726, max timestamp=1680281692740Num uniques 1018; 2023-03-31 16:54:52,743 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-03-31 16:54:52,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:52,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:52,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:52,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:52,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:52,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:52,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:52,746 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:52,746 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:52,746 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:52,746 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:52,746 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:52,746 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:52,748 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:52,748 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:52,748 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:52,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:52,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:52,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:52,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:52,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:52,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:52,748 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:52,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:52,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:52,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:52,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:52,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:52,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:52,750 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:52,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:52,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:52,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:52,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:52,753 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:52,753 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:52,753 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:52,753 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:52,753 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:52,753 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:52,753 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:52,753 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:52,753 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:52,753 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:52,753 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:52,753 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:52,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:52,753 DEBUG [Thread-1666] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:52,753 DEBUG [Thread-1666] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:52,753 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:52,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:52,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:52,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:52,777 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680281692762, max timestamp=1680281692775Num uniques 1077; 2023-03-31 16:54:52,777 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-03-31 16:54:52,777 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680281692762, max timestamp=1680281692775Num uniques 1077; 2023-03-31 16:54:52,777 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-03-31 16:54:52,778 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680281692762, max timestamp=1680281692775Num uniques 1077; 2023-03-31 16:54:52,778 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-03-31 16:54:52,802 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680281692795, max timestamp=1680281692800Num uniques 1086; 2023-03-31 16:54:52,802 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-03-31 16:54:52,802 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680281692795, max timestamp=1680281692800Num uniques 1086; 2023-03-31 16:54:52,802 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680281692795, max timestamp=1680281692800Num uniques 1086; 2023-03-31 16:54:52,803 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-03-31 16:54:52,803 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-03-31 16:54:52,839 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1680281692827, max timestamp=1680281692837Num uniques 1045; 2023-03-31 16:54:52,839 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-03-31 16:54:52,839 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1680281692827, max timestamp=1680281692837Num uniques 1045; 2023-03-31 16:54:52,839 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-03-31 16:54:52,840 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1680281692827, max timestamp=1680281692837Num uniques 1045; 2023-03-31 16:54:52,840 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-03-31 16:54:52,853 DEBUG [Thread-1666] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:52,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:52,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:52,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:52,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:52,864 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:52,864 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:52,864 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:52,864 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:52,864 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:52,865 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:52,866 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:52,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:52,866 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:52,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:52,866 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:52,866 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:52,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:52,867 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:52,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:52,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:52,867 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:52,867 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:52,868 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:52,868 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:52,868 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:52,868 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:52,872 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:52,872 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:52,872 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:52,872 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. started... 2023-03-31 16:54:52,873 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. as already flushing 2023-03-31 16:54:52,873 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:52,873 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:52,873 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:52,873 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:52,873 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:52,873 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:52,874 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:52,874 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:52,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:52,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:52,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:52,875 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:52,875 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:52,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:52,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:52,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:52,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:52,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:52,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:52,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67e67d51[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:52,876 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:52,876 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:52,877 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:52,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:52,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:52,878 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:52,878 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:52,878 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:52,878 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:52,878 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:52,882 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9337, totalHeapSize=21456, min timestamp=1680281692867, max timestamp=1680281692880Num uniques 1104; 2023-03-31 16:54:52,883 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50557, new segment data size=9337, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-03-31 16:54:52,883 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9334, totalHeapSize=21456, min timestamp=1680281692867, max timestamp=1680281692880Num uniques 1104; 2023-03-31 16:54:52,883 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50554, new segment data size=9334, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-03-31 16:54:52,883 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1680281692867, max timestamp=1680281692880Num uniques 1104; 2023-03-31 16:54:52,883 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=9340, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-03-31 16:54:52,885 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:52,885 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:52,885 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:52,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:52,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:52,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:52,885 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:52,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:52,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:52,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:52,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:52,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:52,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:52,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:52,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:52,887 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:52,887 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:52,887 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:52,887 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:52,887 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:52,887 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:52,889 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:52,889 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:52,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:52,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:52,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:52,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:52,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:52,889 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:52,889 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:52,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:52,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:52,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:52,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:52,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:52,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:52,891 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:52,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:52,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:52,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:52,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:52,893 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:52,893 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:52,893 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:52,893 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:52,893 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:52,893 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:52,893 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:52,893 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:52,893 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:52,893 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:52,893 DEBUG [Thread-1666] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:52,893 DEBUG [Thread-1666] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:52,893 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:52,896 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:52,896 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:52,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:52,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:52,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:52,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:52,916 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1680281692904, max timestamp=1680281692913Num uniques 1063; 2023-03-31 16:54:52,916 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1680281692904, max timestamp=1680281692913Num uniques 1063; 2023-03-31 16:54:52,916 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=7458, suffix heap size=138520,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=163 2023-03-31 16:54:52,916 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1680281692904, max timestamp=1680281692913Num uniques 1063; 2023-03-31 16:54:52,916 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=7458, suffix heap size=138520,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=163 2023-03-31 16:54:52,916 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=7458, suffix heap size=138520,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=163 2023-03-31 16:54:52,948 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1680281692940, max timestamp=1680281692945Num uniques 1022; 2023-03-31 16:54:52,948 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1680281692940, max timestamp=1680281692945Num uniques 1022; 2023-03-31 16:54:52,948 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-03-31 16:54:52,948 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1680281692940, max timestamp=1680281692945Num uniques 1022; 2023-03-31 16:54:52,948 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-03-31 16:54:52,948 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-03-31 16:54:52,977 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1680281692967, max timestamp=1680281692974Num uniques 1031; 2023-03-31 16:54:52,977 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1680281692967, max timestamp=1680281692974Num uniques 1031; 2023-03-31 16:54:52,977 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=10576, suffix heap size=135192,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=231 2023-03-31 16:54:52,977 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1680281692967, max timestamp=1680281692974Num uniques 1031; 2023-03-31 16:54:52,977 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=10576, suffix heap size=135192,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=231 2023-03-31 16:54:52,977 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=10576, suffix heap size=135192,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=231 2023-03-31 16:54:52,993 DEBUG [Thread-1666] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:52,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:52,995 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:52,998 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1680281692989, max timestamp=1680281692995Num uniques 1090; 2023-03-31 16:54:52,998 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=8700, suffix heap size=141328,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=190 2023-03-31 16:54:52,998 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1680281692989, max timestamp=1680281692995Num uniques 1090; 2023-03-31 16:54:52,998 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=8700, suffix heap size=141328,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=190 2023-03-31 16:54:52,998 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1680281692989, max timestamp=1680281692995Num uniques 1090; 2023-03-31 16:54:52,998 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=8700, suffix heap size=141328,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=190 2023-03-31 16:54:52,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:52,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:52,999 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:52,999 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:52,999 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:52,999 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:52,999 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:52,999 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:53,000 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:53,000 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:53,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:53,001 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,001 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:53,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:53,001 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:53,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:53,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:53,002 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:53,002 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:53,002 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:53,002 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:53,002 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:53,002 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:53,002 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:53,002 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:53,002 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:53,002 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. started... 2023-03-31 16:54:53,002 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. as already flushing 2023-03-31 16:54:53,002 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:53,002 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:53,003 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:53,003 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,003 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:53,003 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:53,005 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:53,005 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:53,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:53,005 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:53,005 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:53,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:53,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:53,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:53,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:53,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:53,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d1db1ab[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:53,007 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:53,007 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:53,008 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:53,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:53,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:53,008 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:53,008 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,008 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:53,008 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:53,008 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:53,010 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:53,010 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:53,010 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,010 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:53,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:53,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:53,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:53,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:53,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:53,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:53,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:53,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:53,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:53,013 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:53,013 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:53,013 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:53,013 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:53,013 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:53,014 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:53,014 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:53,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:53,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:53,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:53,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:53,014 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:53,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:53,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:53,015 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:53,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:53,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:53,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:53,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:53,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:53,016 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:53,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:53,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,020 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1680281693007, max timestamp=1680281693017Num uniques 1049; 2023-03-31 16:54:53,020 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=11404, suffix heap size=137064,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=249 2023-03-31 16:54:53,020 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1680281693007, max timestamp=1680281693017Num uniques 1049; 2023-03-31 16:54:53,020 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=11404, suffix heap size=137064,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=249 2023-03-31 16:54:53,020 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1680281693007, max timestamp=1680281693017Num uniques 1049; 2023-03-31 16:54:53,020 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=11404, suffix heap size=137064,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=249 2023-03-31 16:54:53,022 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:53,022 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,022 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:53,022 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:53,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:53,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:53,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:53,023 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:53,022 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:53,023 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:53,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:53,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:53,023 DEBUG [Thread-1666] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:53,023 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:53,023 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,023 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:53,023 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:53,023 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:53,023 DEBUG [Thread-1666] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:53,044 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1680281693031, max timestamp=1680281693042Num uniques 1108; 2023-03-31 16:54:53,044 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1680281693031, max timestamp=1680281693042Num uniques 1108; 2023-03-31 16:54:53,044 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=7230, suffix heap size=143200,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=158 2023-03-31 16:54:53,044 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=7230, suffix heap size=143200,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=158 2023-03-31 16:54:53,045 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1680281693031, max timestamp=1680281693042Num uniques 1108; 2023-03-31 16:54:53,045 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=7230, suffix heap size=143200,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=158 2023-03-31 16:54:53,064 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1680281693054, max timestamp=1680281693062Num uniques 1017; 2023-03-31 16:54:53,064 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=9932, suffix heap size=133736,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=217 2023-03-31 16:54:53,064 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1680281693054, max timestamp=1680281693062Num uniques 1017; 2023-03-31 16:54:53,064 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1680281693054, max timestamp=1680281693062Num uniques 1017; 2023-03-31 16:54:53,064 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=9932, suffix heap size=133736,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=217 2023-03-31 16:54:53,064 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=9932, suffix heap size=133736,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=217 2023-03-31 16:54:53,079 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1680281693065, max timestamp=1680281693077Num uniques 1076; 2023-03-31 16:54:53,079 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=8056, suffix heap size=139872,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=176 2023-03-31 16:54:53,079 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1680281693065, max timestamp=1680281693077Num uniques 1076; 2023-03-31 16:54:53,079 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1680281693065, max timestamp=1680281693077Num uniques 1076; 2023-03-31 16:54:53,079 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=8056, suffix heap size=139872,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=176 2023-03-31 16:54:53,080 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=8056, suffix heap size=139872,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=176 2023-03-31 16:54:53,093 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1680281693077, max timestamp=1680281693091Num uniques 1035; 2023-03-31 16:54:53,093 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=10760, suffix heap size=135608,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=235 2023-03-31 16:54:53,093 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1680281693077, max timestamp=1680281693091Num uniques 1035; 2023-03-31 16:54:53,093 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=10760, suffix heap size=135608,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=235 2023-03-31 16:54:53,093 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1680281693077, max timestamp=1680281693091Num uniques 1035; 2023-03-31 16:54:53,094 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=10760, suffix heap size=135608,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=235 2023-03-31 16:54:53,097 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=231 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/B/50886de2d5214df1a70385edb0901e00 2023-03-31 16:54:53,110 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=294, cellSize=13464, totalHeapSize=30816, min timestamp=1680281693103, max timestamp=1680281693108Num uniques 1094; 2023-03-31 16:54:53,110 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=13464, suffix heap size=141744,new segment heap size=30816 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=294 2023-03-31 16:54:53,110 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=294, cellSize=13464, totalHeapSize=30816, min timestamp=1680281693103, max timestamp=1680281693108Num uniques 1094; 2023-03-31 16:54:53,110 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=13464, suffix heap size=141744,new segment heap size=30816 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=294 2023-03-31 16:54:53,111 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=294, cellSize=13464, totalHeapSize=30816, min timestamp=1680281693103, max timestamp=1680281693108Num uniques 1094; 2023-03-31 16:54:53,111 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=13464, suffix heap size=141744,new segment heap size=30816 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=294 2023-03-31 16:54:53,123 DEBUG [Thread-1666] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:53,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:53,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:53,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:53,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:53,127 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:53,127 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:53,127 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:53,127 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:53,127 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:53,127 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:53,128 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:53,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1680281753125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,128 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:53,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:53,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:53,128 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,129 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=303, cellSize=13868, totalHeapSize=31752, min timestamp=1680281693120, max timestamp=1680281693126Num uniques 1153; 2023-03-31 16:54:53,129 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=303, cellSize=13868, totalHeapSize=31752, min timestamp=1680281693120, max timestamp=1680281693126Num uniques 1153; 2023-03-31 16:54:53,129 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,129 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:53,129 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=52798, new segment data size=13868, suffix heap size=147880,new segment heap size=31752 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1153, new segment cells count=303 2023-03-31 16:54:53,129 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=52798, new segment data size=13868, suffix heap size=147880,new segment heap size=31752 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1153, new segment cells count=303 2023-03-31 16:54:53,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:53,129 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=303, cellSize=13868, totalHeapSize=31752, min timestamp=1680281693120, max timestamp=1680281693126Num uniques 1153; 2023-03-31 16:54:53,129 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=52798, new segment data size=13868, suffix heap size=147880,new segment heap size=31752 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1153, new segment cells count=303 2023-03-31 16:54:53,130 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:53,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:53,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:53,130 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:53,130 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:53,130 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:53,130 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:53,130 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:53,130 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:53,130 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:53,131 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:53,131 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:53,131 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. started... 2023-03-31 16:54:53,131 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. as already flushing 2023-03-31 16:54:53,131 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:53,131 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:53,131 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:53,131 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,131 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:53,131 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:53,132 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,132 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:53,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:53,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:53,133 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:53,133 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:53,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:53,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:53,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:53,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:53,134 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:53,134 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17bfadb5[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:53,134 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:53,134 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:53,135 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:53,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:53,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:53,135 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:53,135 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:53,135 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:53,135 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,136 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:53,138 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:53,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:53,138 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:53,138 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:53,138 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:53,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:53,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:53,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:53,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:53,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:53,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:53,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:53,141 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:53,141 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:53,141 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:53,141 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:53,141 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:53,142 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:53,143 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:53,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:53,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:53,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:53,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:53,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:53,143 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:53,143 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:53,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:53,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:53,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:53,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:53,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:53,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:53,144 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:53,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:53,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,146 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,146 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:53,146 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:53,146 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:53,146 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:53,146 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,146 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:53,146 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:53,146 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:53,146 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:53,146 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:53,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:53,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:53,146 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:53,146 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:53,147 DEBUG [Thread-1666] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:53,147 DEBUG [Thread-1666] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:53,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:53,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:53,149 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=212, cellSize=9702, totalHeapSize=22288, min timestamp=1680281693140, max timestamp=1680281693147Num uniques 1162; 2023-03-31 16:54:53,149 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=53212, new segment data size=9702, suffix heap size=148816,new segment heap size=22288 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1162, new segment cells count=212 2023-03-31 16:54:53,150 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=212, cellSize=9702, totalHeapSize=22288, min timestamp=1680281693140, max timestamp=1680281693147Num uniques 1162; 2023-03-31 16:54:53,150 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=53212, new segment data size=9702, suffix heap size=148816,new segment heap size=22288 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1162, new segment cells count=212 2023-03-31 16:54:53,150 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=212, cellSize=9702, totalHeapSize=22288, min timestamp=1680281693140, max timestamp=1680281693147Num uniques 1162; 2023-03-31 16:54:53,150 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=53212, new segment data size=9702, suffix heap size=148816,new segment heap size=22288 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1162, new segment cells count=212 2023-03-31 16:54:53,191 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=171, cellSize=7826, totalHeapSize=18024, min timestamp=1680281693161, max timestamp=1680281693189Num uniques 1071; 2023-03-31 16:54:53,191 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49046, new segment data size=7826, suffix heap size=139352,new segment heap size=18024 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1071, new segment cells count=171 2023-03-31 16:54:53,191 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=171, cellSize=7826, totalHeapSize=18024, min timestamp=1680281693161, max timestamp=1680281693189Num uniques 1071; 2023-03-31 16:54:53,192 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49046, new segment data size=7826, suffix heap size=139352,new segment heap size=18024 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1071, new segment cells count=171 2023-03-31 16:54:53,192 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=171, cellSize=7826, totalHeapSize=18024, min timestamp=1680281693161, max timestamp=1680281693189Num uniques 1071; 2023-03-31 16:54:53,192 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49046, new segment data size=7826, suffix heap size=139352,new segment heap size=18024 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1071, new segment cells count=171 2023-03-31 16:54:53,212 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=230, cellSize=10530, totalHeapSize=24160, min timestamp=1680281693198, max timestamp=1680281693210Num uniques 1030; 2023-03-31 16:54:53,212 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47170, new segment data size=10530, suffix heap size=135088,new segment heap size=24160 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1030, new segment cells count=230 2023-03-31 16:54:53,212 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=230, cellSize=10530, totalHeapSize=24160, min timestamp=1680281693198, max timestamp=1680281693210Num uniques 1030; 2023-03-31 16:54:53,212 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=230, cellSize=10530, totalHeapSize=24160, min timestamp=1680281693198, max timestamp=1680281693210Num uniques 1030; 2023-03-31 16:54:53,212 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47170, new segment data size=10530, suffix heap size=135088,new segment heap size=24160 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1030, new segment cells count=230 2023-03-31 16:54:53,212 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47170, new segment data size=10530, suffix heap size=135088,new segment heap size=24160 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1030, new segment cells count=230 2023-03-31 16:54:53,249 DEBUG [Thread-1666] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:53,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:53,251 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:53,254 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=239, cellSize=10944, totalHeapSize=25096, min timestamp=1680281693224, max timestamp=1680281693251Num uniques 1089; 2023-03-31 16:54:53,254 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49874, new segment data size=10944, suffix heap size=141224,new segment heap size=25096 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1089, new segment cells count=239 2023-03-31 16:54:53,254 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=239, cellSize=10944, totalHeapSize=25096, min timestamp=1680281693224, max timestamp=1680281693251Num uniques 1089; 2023-03-31 16:54:53,254 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49874, new segment data size=10944, suffix heap size=141224,new segment heap size=25096 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1089, new segment cells count=239 2023-03-31 16:54:53,254 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=239, cellSize=10944, totalHeapSize=25096, min timestamp=1680281693224, max timestamp=1680281693251Num uniques 1089; 2023-03-31 16:54:53,254 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49874, new segment data size=10944, suffix heap size=141224,new segment heap size=25096 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1089, new segment cells count=239 2023-03-31 16:54:53,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:53,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:53,255 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:53,255 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:53,255 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:53,255 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:53,255 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:53,255 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:53,257 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:53,257 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:53,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:53,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:53,258 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,258 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:53,258 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:53,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:53,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:53,258 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:53,258 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:53,258 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:53,259 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:53,259 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:53,259 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:53,259 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:53,259 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:53,259 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. started... 2023-03-31 16:54:53,260 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. as already flushing 2023-03-31 16:54:53,260 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:53,260 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:53,260 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:53,260 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,260 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:53,260 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:53,262 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:53,262 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:53,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:53,263 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:53,263 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:53,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:53,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:53,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:53,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:53,264 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:53,264 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:53,264 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@513d4264[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:53,265 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:53,266 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:53,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:53,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:53,266 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:53,266 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:53,266 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:53,266 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,266 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:53,267 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:53,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:53,268 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:53,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:53,268 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:53,268 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:53,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:53,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:53,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:53,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:53,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:53,274 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:53,274 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:53,274 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:53,274 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:53,274 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:53,274 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:53,276 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=198, cellSize=9068, totalHeapSize=20832, min timestamp=1680281693265, max timestamp=1680281693274Num uniques 1098; 2023-03-31 16:54:53,276 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50288, new segment data size=9068, suffix heap size=142160,new segment heap size=20832 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1098, new segment cells count=198 2023-03-31 16:54:53,277 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=198, cellSize=9068, totalHeapSize=20832, min timestamp=1680281693265, max timestamp=1680281693274Num uniques 1098; 2023-03-31 16:54:53,277 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50288, new segment data size=9068, suffix heap size=142160,new segment heap size=20832 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1098, new segment cells count=198 2023-03-31 16:54:53,277 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=198, cellSize=9068, totalHeapSize=20832, min timestamp=1680281693265, max timestamp=1680281693274Num uniques 1098; 2023-03-31 16:54:53,277 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50288, new segment data size=9068, suffix heap size=142160,new segment heap size=20832 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1098, new segment cells count=198 2023-03-31 16:54:53,278 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:53,278 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:53,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:53,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:53,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:53,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:53,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:53,278 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:53,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:53,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:53,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:53,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:53,279 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:53,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:53,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:53,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,281 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:53,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:53,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,284 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,284 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:53,284 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:53,284 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,284 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:53,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:53,284 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:53,284 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:53,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:53,284 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:53,284 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:53,284 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:53,284 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:53,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:53,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:53,284 DEBUG [Thread-1666] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:53,285 DEBUG [Thread-1666] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:53,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:53,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:53,299 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=257, cellSize=11765, totalHeapSize=26968, min timestamp=1680281693282, max timestamp=1680281693297Num uniques 1057; 2023-03-31 16:54:53,299 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=257, cellSize=11765, totalHeapSize=26968, min timestamp=1680281693282, max timestamp=1680281693297Num uniques 1057; 2023-03-31 16:54:53,299 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=257, cellSize=11765, totalHeapSize=26968, min timestamp=1680281693282, max timestamp=1680281693297Num uniques 1057; 2023-03-31 16:54:53,299 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48405, new segment data size=11765, suffix heap size=137896,new segment heap size=26968 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1057, new segment cells count=257 2023-03-31 16:54:53,300 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48405, new segment data size=11765, suffix heap size=137896,new segment heap size=26968 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1057, new segment cells count=257 2023-03-31 16:54:53,299 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48405, new segment data size=11765, suffix heap size=137896,new segment heap size=26968 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1057, new segment cells count=257 2023-03-31 16:54:53,316 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:53,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1680281753313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,317 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=216, cellSize=9883, totalHeapSize=22704, min timestamp=1680281693310, max timestamp=1680281693315Num uniques 1116; 2023-03-31 16:54:53,317 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=51103, new segment data size=9883, suffix heap size=144032,new segment heap size=22704 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1116, new segment cells count=216 2023-03-31 16:54:53,318 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=216, cellSize=9880, totalHeapSize=22704, min timestamp=1680281693310, max timestamp=1680281693315Num uniques 1116; 2023-03-31 16:54:53,318 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=216, cellSize=9883, totalHeapSize=22704, min timestamp=1680281693310, max timestamp=1680281693315Num uniques 1116; 2023-03-31 16:54:53,318 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=51100, new segment data size=9880, suffix heap size=144032,new segment heap size=22704 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1116, new segment cells count=216 2023-03-31 16:54:53,318 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=51103, new segment data size=9883, suffix heap size=144032,new segment heap size=22704 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1116, new segment cells count=216 2023-03-31 16:54:53,339 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=225, cellSize=10300, totalHeapSize=23640, min timestamp=1680281693332, max timestamp=1680281693337Num uniques 1075; 2023-03-31 16:54:53,339 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49230, new segment data size=10300, suffix heap size=139768,new segment heap size=23640 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1075, new segment cells count=225 2023-03-31 16:54:53,340 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=225, cellSize=10300, totalHeapSize=23640, min timestamp=1680281693332, max timestamp=1680281693337Num uniques 1075; 2023-03-31 16:54:53,340 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49230, new segment data size=10300, suffix heap size=139768,new segment heap size=23640 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1075, new segment cells count=225 2023-03-31 16:54:53,340 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=225, cellSize=10300, totalHeapSize=23640, min timestamp=1680281693332, max timestamp=1680281693337Num uniques 1075; 2023-03-31 16:54:53,340 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49230, new segment data size=10300, suffix heap size=139768,new segment heap size=23640 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1075, new segment cells count=225 2023-03-31 16:54:53,357 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=234, cellSize=10714, totalHeapSize=24576, min timestamp=1680281693351, max timestamp=1680281693355Num uniques 1084; 2023-03-31 16:54:53,357 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49644, new segment data size=10714, suffix heap size=140704,new segment heap size=24576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1084, new segment cells count=234 2023-03-31 16:54:53,358 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=234, cellSize=10714, totalHeapSize=24576, min timestamp=1680281693351, max timestamp=1680281693355Num uniques 1084; 2023-03-31 16:54:53,358 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=234, cellSize=10714, totalHeapSize=24576, min timestamp=1680281693351, max timestamp=1680281693355Num uniques 1084; 2023-03-31 16:54:53,358 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49644, new segment data size=10714, suffix heap size=140704,new segment heap size=24576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1084, new segment cells count=234 2023-03-31 16:54:53,358 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49644, new segment data size=10714, suffix heap size=140704,new segment heap size=24576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1084, new segment cells count=234 2023-03-31 16:54:53,379 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=243, cellSize=11118, totalHeapSize=25512, min timestamp=1680281693371, max timestamp=1680281693376Num uniques 1093; 2023-03-31 16:54:53,379 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50048, new segment data size=11118, suffix heap size=141640,new segment heap size=25512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1093, new segment cells count=243 2023-03-31 16:54:53,379 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=243, cellSize=11118, totalHeapSize=25512, min timestamp=1680281693371, max timestamp=1680281693376Num uniques 1093; 2023-03-31 16:54:53,380 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=243, cellSize=11118, totalHeapSize=25512, min timestamp=1680281693371, max timestamp=1680281693376Num uniques 1093; 2023-03-31 16:54:53,380 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50048, new segment data size=11118, suffix heap size=141640,new segment heap size=25512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1093, new segment cells count=243 2023-03-31 16:54:53,380 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50048, new segment data size=11118, suffix heap size=141640,new segment heap size=25512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1093, new segment cells count=243 2023-03-31 16:54:53,385 DEBUG [Thread-1666] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:53,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:53,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:53,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:53,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:53,388 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:53,388 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:53,388 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:53,388 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:53,388 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:53,388 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:53,389 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,389 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:53,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:53,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:53,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:53,390 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,390 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:53,390 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:53,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:53,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:53,391 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:53,391 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:53,391 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:53,391 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:53,391 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:53,391 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:53,391 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:53,391 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:53,391 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:53,391 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. started... 2023-03-31 16:54:53,391 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. as already flushing 2023-03-31 16:54:53,391 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:53,392 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:53,392 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:53,392 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,392 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:53,392 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:53,393 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:53,393 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:53,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:53,393 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:53,393 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:53,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:53,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:53,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:53,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:53,395 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:53,395 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@565a1e7[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:53,395 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:53,395 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:53,396 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:53,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:53,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:53,396 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:53,396 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:53,396 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:53,396 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,396 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:53,398 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,398 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:53,398 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:53,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:53,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:53,398 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:53,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:53,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:53,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:53,401 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=152, cellSize=6960, totalHeapSize=16048, min timestamp=1680281693393, max timestamp=1680281693398Num uniques 1102; 2023-03-31 16:54:53,401 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=152, cellSize=6960, totalHeapSize=16048, min timestamp=1680281693393, max timestamp=1680281693398Num uniques 1102; 2023-03-31 16:54:53,401 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50470, new segment data size=6960, suffix heap size=142576,new segment heap size=16048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1102, new segment cells count=152 2023-03-31 16:54:53,401 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=152, cellSize=6960, totalHeapSize=16048, min timestamp=1680281693393, max timestamp=1680281693398Num uniques 1102; 2023-03-31 16:54:53,401 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50470, new segment data size=6960, suffix heap size=142576,new segment heap size=16048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1102, new segment cells count=152 2023-03-31 16:54:53,401 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50470, new segment data size=6960, suffix heap size=142576,new segment heap size=16048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1102, new segment cells count=152 2023-03-31 16:54:53,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:53,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:53,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:53,403 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:53,403 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:53,403 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:53,403 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:53,403 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:53,403 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:53,404 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:53,404 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:53,404 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:53,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:53,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:53,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:53,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:53,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:53,405 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:53,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:53,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:53,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:53,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:53,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:53,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:53,406 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:53,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:53,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,408 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,408 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:53,409 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:53,409 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,409 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:53,409 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:53,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:53,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:53,409 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:53,409 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:53,409 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:53,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:53,409 DEBUG [Thread-1666] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:53,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:53,409 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:53,409 DEBUG [Thread-1666] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:53,409 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:53,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:53,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:53,424 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=211, cellSize=9656, totalHeapSize=22184, min timestamp=1680281693415, max timestamp=1680281693423Num uniques 1011; 2023-03-31 16:54:53,424 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46296, new segment data size=9656, suffix heap size=133112,new segment heap size=22184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1011, new segment cells count=211 2023-03-31 16:54:53,425 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=211, cellSize=9656, totalHeapSize=22184, min timestamp=1680281693415, max timestamp=1680281693423Num uniques 1011; 2023-03-31 16:54:53,425 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46296, new segment data size=9656, suffix heap size=133112,new segment heap size=22184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1011, new segment cells count=211 2023-03-31 16:54:53,425 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=211, cellSize=9656, totalHeapSize=22184, min timestamp=1680281693415, max timestamp=1680281693423Num uniques 1011; 2023-03-31 16:54:53,425 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46296, new segment data size=9656, suffix heap size=133112,new segment heap size=22184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1011, new segment cells count=211 2023-03-31 16:54:53,441 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=270, cellSize=12360, totalHeapSize=28320, min timestamp=1680281693430, max timestamp=1680281693439Num uniques 1070; 2023-03-31 16:54:53,441 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49000, new segment data size=12360, suffix heap size=139248,new segment heap size=28320 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1070, new segment cells count=270 2023-03-31 16:54:53,441 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=270, cellSize=12360, totalHeapSize=28320, min timestamp=1680281693430, max timestamp=1680281693439Num uniques 1070; 2023-03-31 16:54:53,441 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49000, new segment data size=12360, suffix heap size=139248,new segment heap size=28320 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1070, new segment cells count=270 2023-03-31 16:54:53,441 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=270, cellSize=12360, totalHeapSize=28320, min timestamp=1680281693430, max timestamp=1680281693439Num uniques 1070; 2023-03-31 16:54:53,442 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49000, new segment data size=12360, suffix heap size=139248,new segment heap size=28320 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1070, new segment cells count=270 2023-03-31 16:54:53,457 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=229, cellSize=10484, totalHeapSize=24056, min timestamp=1680281693450, max timestamp=1680281693455Num uniques 1129; 2023-03-31 16:54:53,457 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=229, cellSize=10484, totalHeapSize=24056, min timestamp=1680281693450, max timestamp=1680281693455Num uniques 1129; 2023-03-31 16:54:53,457 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=51704, new segment data size=10484, suffix heap size=145384,new segment heap size=24056 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1129, new segment cells count=229 2023-03-31 16:54:53,457 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=51704, new segment data size=10484, suffix heap size=145384,new segment heap size=24056 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1129, new segment cells count=229 2023-03-31 16:54:53,457 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=229, cellSize=10484, totalHeapSize=24056, min timestamp=1680281693450, max timestamp=1680281693455Num uniques 1129; 2023-03-31 16:54:53,457 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=51704, new segment data size=10484, suffix heap size=145384,new segment heap size=24056 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1129, new segment cells count=229 2023-03-31 16:54:53,475 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=238, cellSize=10898, totalHeapSize=24992, min timestamp=1680281693465, max timestamp=1680281693472Num uniques 1088; 2023-03-31 16:54:53,475 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=238, cellSize=10898, totalHeapSize=24992, min timestamp=1680281693465, max timestamp=1680281693472Num uniques 1088; 2023-03-31 16:54:53,475 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=238, cellSize=10898, totalHeapSize=24992, min timestamp=1680281693465, max timestamp=1680281693472Num uniques 1088; 2023-03-31 16:54:53,475 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49828, new segment data size=10898, suffix heap size=141120,new segment heap size=24992 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1088, new segment cells count=238 2023-03-31 16:54:53,475 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49828, new segment data size=10898, suffix heap size=141120,new segment heap size=24992 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1088, new segment cells count=238 2023-03-31 16:54:53,475 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49828, new segment data size=10898, suffix heap size=141120,new segment heap size=24992 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1088, new segment cells count=238 2023-03-31 16:54:53,506 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=231 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/C/e0026de07c424e348a0b7ebdbd77ebfa 2023-03-31 16:54:53,509 DEBUG [Thread-1666] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:53,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:53,510 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/A/613a872127fb4c298ce182599b1d6a61 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/613a872127fb4c298ce182599b1d6a61 2023-03-31 16:54:53,513 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:53,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:53,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:53,514 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:53,514 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:53,514 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:53,514 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:53,515 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:53,515 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:53,516 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/613a872127fb4c298ce182599b1d6a61, entries=150, sequenceid=231, filesize=11.8 K 2023-03-31 16:54:53,516 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/B/50886de2d5214df1a70385edb0901e00 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/50886de2d5214df1a70385edb0901e00 2023-03-31 16:54:53,520 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:53,520 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:53,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40794 deadline: 1680281753492, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40748 deadline: 1680281753491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,521 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=197, cellSize=9002, totalHeapSize=20728, min timestamp=1680281693487, max timestamp=1680281693499Num uniques 1097; 2023-03-31 16:54:53,521 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50222, new segment data size=9002, suffix heap size=142056,new segment heap size=20728 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1097, new segment cells count=197 2023-03-31 16:54:53,521 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=183, cellSize=8368, totalHeapSize=19272, min timestamp=1680281693487, max timestamp=1680281693499Num uniques 1097; 2023-03-31 16:54:53,521 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50222, new segment data size=8368, suffix heap size=142056,new segment heap size=19272 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1097, new segment cells count=183 2023-03-31 16:54:53,521 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:53,521 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=197, cellSize=9002, totalHeapSize=20728, min timestamp=1680281693487, max timestamp=1680281693499Num uniques 1097; 2023-03-31 16:54:53,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:53,521 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50222, new segment data size=9002, suffix heap size=142056,new segment heap size=20728 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1097, new segment cells count=197 2023-03-31 16:54:53,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:53,522 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:53,522 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,522 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:53,522 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:53,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:53,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:53,523 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:53,523 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:53,523 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:53,523 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:53,523 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:53,523 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:53,523 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:53,523 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:53,524 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:53,524 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/50886de2d5214df1a70385edb0901e00, entries=150, sequenceid=231, filesize=11.8 K 2023-03-31 16:54:53,524 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. started... 2023-03-31 16:54:53,524 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. as already flushing 2023-03-31 16:54:53,524 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:53,524 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:53,524 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:53,524 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,524 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:53,524 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:53,525 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/C/e0026de07c424e348a0b7ebdbd77ebfa as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/e0026de07c424e348a0b7ebdbd77ebfa 2023-03-31 16:54:53,526 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,526 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:53,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:53,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:53,526 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:53,527 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:53,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:53,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:53,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:53,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:53,528 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/e0026de07c424e348a0b7ebdbd77ebfa, entries=150, sequenceid=231, filesize=11.8 K 2023-03-31 16:54:53,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:53,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35c5c809[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:53,529 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:53,529 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:53,529 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~63.66 KB/65184, currentSize=46.34 KB/47456 for 3ab680f7c89c77c9be754a4581df5bc8 in 854ms, sequenceid=231, compaction requested=true 2023-03-31 16:54:53,529 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3ab680f7c89c77c9be754a4581df5bc8: 2023-03-31 16:54:53,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on 3ab680f7c89c77c9be754a4581df5bc8 2023-03-31 16:54:53,529 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 16:54:53,530 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:53,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:53,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:53,530 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:53,530 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:53,530 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,530 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:53,531 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45806 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 16:54:53,531 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1912): 3ab680f7c89c77c9be754a4581df5bc8/A is initiating minor compaction (all files) 2023-03-31 16:54:53,531 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3ab680f7c89c77c9be754a4581df5bc8/A in TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:53,531 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/6e2435bd4c00454db912d5779bcdb3de, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/635d2c14218f4906bfe16bb619d60aa8, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/e35bb37161cb489ba3eea7d527b760fe, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/613a872127fb4c298ce182599b1d6a61] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp, totalSize=44.7 K 2023-03-31 16:54:53,529 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:54:53,530 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:53,531 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 6e2435bd4c00454db912d5779bcdb3de, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=109, earliestPutTs=1680281690713 2023-03-31 16:54:53,531 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 16:54:53,531 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:54:53,531 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:54:53,531 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3ab680f7c89c77c9be754a4581df5bc8 3/3 column families, dataSize=59.76 KB heapSize=160.86 KB 2023-03-31 16:54:53,531 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 635d2c14218f4906bfe16bb619d60aa8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=147, earliestPutTs=1680281690825 2023-03-31 16:54:53,532 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=A 2023-03-31 16:54:53,532 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:54:53,532 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=B 2023-03-31 16:54:53,532 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:54:53,532 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=C 2023-03-31 16:54:53,532 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:54:53,532 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting e35bb37161cb489ba3eea7d527b760fe, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=159, earliestPutTs=1680281691617 2023-03-31 16:54:53,532 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:53,532 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:53,533 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:53,532 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 613a872127fb4c298ce182599b1d6a61, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=231, earliestPutTs=1680281692355 2023-03-31 16:54:53,533 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45806 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 16:54:53,533 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): 3ab680f7c89c77c9be754a4581df5bc8/B is initiating minor compaction (all files) 2023-03-31 16:54:53,533 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3ab680f7c89c77c9be754a4581df5bc8/B in TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:53,533 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/e997c7a82ddb4f5ea3121e9784db51ab, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/9717f3a6d39e4fc1975721d3169d0ca3, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/bd9851dfc85a48a7b39c35ae34a368a1, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/50886de2d5214df1a70385edb0901e00] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp, totalSize=44.7 K 2023-03-31 16:54:53,533 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting e997c7a82ddb4f5ea3121e9784db51ab, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=109, earliestPutTs=1680281690713 2023-03-31 16:54:53,534 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 9717f3a6d39e4fc1975721d3169d0ca3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=147, earliestPutTs=1680281690825 2023-03-31 16:54:53,534 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting bd9851dfc85a48a7b39c35ae34a368a1, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=159, earliestPutTs=1680281691617 2023-03-31 16:54:53,534 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,534 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 50886de2d5214df1a70385edb0901e00, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=231, earliestPutTs=1680281692355 2023-03-31 16:54:53,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:53,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:53,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:53,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:53,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:53,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:53,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:53,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:53,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:53,538 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:53,538 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:53,538 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:53,538 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:53,538 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:53,542 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:53,542 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3ab680f7c89c77c9be754a4581df5bc8#A#compaction#237 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:54:53,542 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:53,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:53,542 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:53,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:53,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:53,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:53,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:53,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:53,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:53,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:53,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:53,546 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:53,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:53,546 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): 3ab680f7c89c77c9be754a4581df5bc8#B#compaction#238 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:54:53,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:53,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,547 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:53,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:53,548 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=806 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/A/7a38f1c9abf54c96b1180fda896def39 2023-03-31 16:54:53,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,549 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:53,549 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:53,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:53,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:53,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:53,549 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,550 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:53,550 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:53,550 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,550 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:53,550 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:53,550 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:53,549 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:53,549 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:53,550 DEBUG [Thread-1666] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:53,550 DEBUG [Thread-1666] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:53,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:53,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:53,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:53,561 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/A/654233eeecf943a4bb2d01ba2861d573 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/654233eeecf943a4bb2d01ba2861d573 2023-03-31 16:54:53,563 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=806 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/B/e3bfb109cc3e413d865bc65e82158419 2023-03-31 16:54:53,568 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3ab680f7c89c77c9be754a4581df5bc8/A of 3ab680f7c89c77c9be754a4581df5bc8 into 654233eeecf943a4bb2d01ba2861d573(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:54:53,568 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3ab680f7c89c77c9be754a4581df5bc8: 2023-03-31 16:54:53,568 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8., storeName=3ab680f7c89c77c9be754a4581df5bc8/A, priority=12, startTime=1680281693529; duration=0sec 2023-03-31 16:54:53,568 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:53,569 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:54:53,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1680281753563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,569 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 16:54:53,570 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1680281693557, max timestamp=1680281693566Num uniques 859; 2023-03-31 16:54:53,570 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 16:54:53,570 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1680281693557, max timestamp=1680281693566Num uniques 859; 2023-03-31 16:54:53,570 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 16:54:53,570 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1680281693557, max timestamp=1680281693566Num uniques 859; 2023-03-31 16:54:53,570 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 16:54:53,578 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45806 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 16:54:53,578 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1912): 3ab680f7c89c77c9be754a4581df5bc8/C is initiating minor compaction (all files) 2023-03-31 16:54:53,578 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3ab680f7c89c77c9be754a4581df5bc8/C in TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:53,579 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/fa817bcbf06040269857bfc06c6f8a62, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/5302cd4baa30418996ac4ea47c688c8e, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/48ee8e530bf848169a416a1ad8cb40dc, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/e0026de07c424e348a0b7ebdbd77ebfa] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp, totalSize=44.7 K 2023-03-31 16:54:53,579 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting fa817bcbf06040269857bfc06c6f8a62, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=109, earliestPutTs=1680281690713 2023-03-31 16:54:53,579 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 5302cd4baa30418996ac4ea47c688c8e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=147, earliestPutTs=1680281690825 2023-03-31 16:54:53,580 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 48ee8e530bf848169a416a1ad8cb40dc, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=159, earliestPutTs=1680281691617 2023-03-31 16:54:53,580 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting e0026de07c424e348a0b7ebdbd77ebfa, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=231, earliestPutTs=1680281692355 2023-03-31 16:54:53,587 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=19.51 KB at sequenceid=806 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/C/db96159877b349c989a32da1e10184f6 2023-03-31 16:54:53,590 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3ab680f7c89c77c9be754a4581df5bc8#C#compaction#241 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:54:53,593 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/A/7a38f1c9abf54c96b1180fda896def39 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/7a38f1c9abf54c96b1180fda896def39 2023-03-31 16:54:53,597 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/7a38f1c9abf54c96b1180fda896def39, entries=150, sequenceid=806, filesize=11.9 K 2023-03-31 16:54:53,601 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/B/e3bfb109cc3e413d865bc65e82158419 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/e3bfb109cc3e413d865bc65e82158419 2023-03-31 16:54:53,605 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/e3bfb109cc3e413d865bc65e82158419, entries=150, sequenceid=806, filesize=11.9 K 2023-03-31 16:54:53,607 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/C/db96159877b349c989a32da1e10184f6 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/db96159877b349c989a32da1e10184f6 2023-03-31 16:54:53,608 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/C/2d04dee6d0f94e379b9e1980afdb654a as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/2d04dee6d0f94e379b9e1980afdb654a 2023-03-31 16:54:53,612 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/db96159877b349c989a32da1e10184f6, entries=150, sequenceid=806, filesize=11.9 K 2023-03-31 16:54:53,612 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~59.76 KB/61196, heapSize ~160.81 KB/164672, currentSize=127.47 KB/130530 for 3ab680f7c89c77c9be754a4581df5bc8 in 81ms, sequenceid=806, compaction requested=false 2023-03-31 16:54:53,612 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3ab680f7c89c77c9be754a4581df5bc8: 2023-03-31 16:54:53,617 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3ab680f7c89c77c9be754a4581df5bc8/C of 3ab680f7c89c77c9be754a4581df5bc8 into 2d04dee6d0f94e379b9e1980afdb654a(size=12.1 K), total size for store is 24.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:54:53,617 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3ab680f7c89c77c9be754a4581df5bc8: 2023-03-31 16:54:53,617 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8., storeName=3ab680f7c89c77c9be754a4581df5bc8/C, priority=12, startTime=1680281693531; duration=0sec 2023-03-31 16:54:53,617 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:54:53,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on 3ab680f7c89c77c9be754a4581df5bc8 2023-03-31 16:54:53,618 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3ab680f7c89c77c9be754a4581df5bc8 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-03-31 16:54:53,618 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=A 2023-03-31 16:54:53,618 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:54:53,618 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=B 2023-03-31 16:54:53,618 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:54:53,618 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=C 2023-03-31 16:54:53,618 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:54:53,629 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:53,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40794 deadline: 1680281753624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,629 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:53,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40748 deadline: 1680281753625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,632 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:53,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40768 deadline: 1680281753628, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,632 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:53,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40716 deadline: 1680281753629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,650 DEBUG [Thread-1666] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:53,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:53,653 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:53,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:53,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:53,657 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:53,657 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:53,657 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:53,657 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:53,657 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:53,657 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:53,658 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,658 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:53,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:53,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:53,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:53,659 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,659 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:53,660 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:53,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:53,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:53,660 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:53,660 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:53,661 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:53,661 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:53,661 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:53,661 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:53,661 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:53,661 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:53,661 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:53,661 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. started... 2023-03-31 16:54:53,661 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. as already flushing 2023-03-31 16:54:53,661 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:53,661 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:53,661 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:53,661 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,661 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:53,661 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:53,663 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,663 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:53,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:53,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:53,663 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:53,664 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:53,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:53,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:53,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:53,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:53,666 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:53,666 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f7d0901[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:53,666 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:53,666 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:53,668 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:53,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:53,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:53,668 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:53,668 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:53,668 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:53,668 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,669 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:53,670 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:53,670 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:53,670 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,670 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:53,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:53,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:53,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:53,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:53,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:53,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:53,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:53,673 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:53,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1680281753671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:53,674 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:53,674 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:53,674 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:53,674 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:53,674 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:53,674 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:53,675 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:53,676 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:53,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:53,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:53,676 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:53,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:53,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:53,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:53,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:53,676 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:53,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:53,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:53,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:53,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:53,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:53,680 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:53,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:53,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,683 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,683 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:53,683 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:53,683 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:53,683 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:53,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:53,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:53,684 DEBUG [Thread-1666] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:53,684 DEBUG [Thread-1666] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:53,683 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:53,683 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:53,684 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:53,684 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,684 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:53,684 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:53,684 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:53,684 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:53,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:53,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:53,732 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:53,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40794 deadline: 1680281753730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,732 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:53,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40748 deadline: 1680281753730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,734 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:53,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40768 deadline: 1680281753733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,735 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:53,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40716 deadline: 1680281753733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,784 DEBUG [Thread-1666] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:53,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:53,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:53,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:53,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:53,787 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:53,787 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:53,787 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:53,787 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:53,788 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:53,788 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:53,789 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,789 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:53,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:53,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:53,789 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,789 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:53,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:53,790 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:53,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:53,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:53,790 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:53,790 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:53,791 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:53,791 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:53,791 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:53,791 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:53,791 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:53,791 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:53,791 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:53,791 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. started... 2023-03-31 16:54:53,791 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. as already flushing 2023-03-31 16:54:53,791 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:53,791 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:53,791 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:53,791 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,791 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:53,791 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:53,793 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,793 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:53,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:53,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:53,793 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:53,793 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:53,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:53,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:53,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:53,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:53,794 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:53,794 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64685ee8[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:53,794 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:53,794 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:53,795 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:53,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:53,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:53,796 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:53,796 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:53,796 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:53,796 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,796 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:53,797 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,797 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:53,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:53,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:53,797 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:53,798 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:53,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:53,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:53,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:53,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:53,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:53,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:53,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:53,800 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:53,800 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:53,801 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:53,801 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:53,801 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:53,806 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:53,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:53,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:53,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:53,807 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:53,807 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:53,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:53,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:53,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:53,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:53,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:53,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:53,808 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:53,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:53,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:53,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,812 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:53,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:53,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,815 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:53,815 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:53,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:53,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:53,816 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,816 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:53,816 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:53,816 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,816 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:53,816 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:53,816 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:53,816 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:53,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:53,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:53,816 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:53,816 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:53,816 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:53,817 DEBUG [Thread-1666] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:53,817 DEBUG [Thread-1666] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:53,878 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:53,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1680281753875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,917 DEBUG [Thread-1666] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:53,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:53,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:53,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:53,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:53,920 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:53,920 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:53,920 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:53,920 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:53,920 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:53,921 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:53,923 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,923 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:53,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:53,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:53,923 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,923 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:53,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:53,924 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:53,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:53,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:53,924 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:53,924 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:53,924 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:53,924 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:53,924 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:53,924 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:53,925 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:53,925 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:53,925 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:53,925 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. started... 2023-03-31 16:54:53,925 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. as already flushing 2023-03-31 16:54:53,925 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:53,925 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:53,925 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:53,925 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,925 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:53,925 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:53,929 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:53,929 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:53,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:53,930 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:53,930 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:53,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:53,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:53,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:53,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:53,931 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:53,931 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4caab11b[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:53,931 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:53,931 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:53,932 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:53,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:53,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:53,932 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:53,932 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:53,932 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:53,933 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,933 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:53,935 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:53,935 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,935 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:53,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:53,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:53,935 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:53,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:53,935 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:53,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40794 deadline: 1680281753933, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:53,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:53,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:53,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:53,937 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:53,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40748 deadline: 1680281753934, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,937 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:53,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40716 deadline: 1680281753935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:53,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:53,938 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:53,938 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:53,938 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:53,938 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:53,938 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:53,938 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:53,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40768 deadline: 1680281753936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,940 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:53,940 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:53,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:53,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:53,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:53,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:53,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:53,940 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:53,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:53,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:53,940 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:53,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:53,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:53,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:53,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:53,941 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:53,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:53,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,944 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,944 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:53,944 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:53,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:53,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:53,944 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:53,944 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:53,944 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:53,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:53,944 DEBUG [Thread-1666] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:53,944 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:53,944 DEBUG [Thread-1666] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:53,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:53,944 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:53,944 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:53,944 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:53,944 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:53,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:53,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:53,957 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/B/12ea4e7eff5d4b728065004a29cfc554 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/12ea4e7eff5d4b728065004a29cfc554 2023-03-31 16:54:53,961 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3ab680f7c89c77c9be754a4581df5bc8/B of 3ab680f7c89c77c9be754a4581df5bc8 into 12ea4e7eff5d4b728065004a29cfc554(size=12.1 K), total size for store is 24.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:54:53,961 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3ab680f7c89c77c9be754a4581df5bc8: 2023-03-31 16:54:53,961 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8., storeName=3ab680f7c89c77c9be754a4581df5bc8/B, priority=12, startTime=1680281693531; duration=0sec 2023-03-31 16:54:53,961 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:54:54,040 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=844 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/A/2c2aee8a7bdf467789516978cd0be6f7 2023-03-31 16:54:54,044 DEBUG [Thread-1666] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:54,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:54,050 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:54,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:54,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:54,056 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:54,056 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:54,056 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:54,056 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:54,057 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:54,057 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:54,058 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=844 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/B/ca44fadb4f7d4ee0b171ca6a1d9c8035 2023-03-31 16:54:54,058 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:54,058 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:54,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:54,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:54,058 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:54,058 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:54,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:54,059 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:54,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:54,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:54,060 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:54,060 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:54,060 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:54,060 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:54,060 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:54,060 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:54,060 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:54,060 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:54,060 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:54,061 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. started... 2023-03-31 16:54:54,061 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. as already flushing 2023-03-31 16:54:54,061 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:54,061 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:54,061 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:54,061 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:54,061 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:54,061 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:54,063 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:54,063 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:54,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:54,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:54,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:54,064 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:54,064 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:54,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:54,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:54,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:54,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:54,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:54,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:54,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57b4bd07[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:54,065 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:54,065 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:54,067 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:54,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:54,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:54,067 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:54,067 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:54,067 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:54,067 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:54,067 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:54,068 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:54,068 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:54,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:54,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:54,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:54,068 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:54,068 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:54,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:54,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:54,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:54,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:54,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:54,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:54,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:54,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:54,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:54,071 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:54,071 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:54,071 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:54,071 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:54,071 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:54,072 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:54,072 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:54,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:54,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:54,072 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:54,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:54,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:54,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:54,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:54,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:54,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:54,073 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:54,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:54,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:54,074 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:54,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:54,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:54,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:54,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:54,077 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:54,077 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:54,077 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:54,077 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:54,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:54,077 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:54,077 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:54,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:54,077 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:54,077 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:54,077 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:54,077 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:54,078 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:54,078 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:54,078 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:54,078 DEBUG [Thread-1666] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:54,078 DEBUG [Thread-1666] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:54,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:54,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:54,178 DEBUG [Thread-1666] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:54,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:54,180 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:54,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:54,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:54,181 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:54,181 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:54,181 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:54,181 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:54,181 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:54,181 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:54,182 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:54,182 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:54,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:54,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:54,183 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:54,183 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:54,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:54,183 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:54,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:54,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:54,183 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:54,184 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:54,184 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:54,184 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:54,184 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:54,184 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:54,184 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:54,184 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:54,184 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:54,184 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. started... 2023-03-31 16:54:54,184 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:54,184 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. as already flushing 2023-03-31 16:54:54,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1680281754181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:54,184 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:54,185 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:54,185 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:54,185 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:54,185 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:54,185 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:54,188 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:54,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:54,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:54,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:54,188 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:54,198 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:54,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:54,198 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:54,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:54,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:54,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:54,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:54,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:54,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60b693f1[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:54,200 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:54,201 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:54,202 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:54,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:54,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:54,202 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:54,202 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:54,202 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:54,202 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:54,202 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:54,204 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:54,204 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:54,204 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:54,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:54,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:54,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:54,204 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:54,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:54,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:54,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:54,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:54,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:54,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:54,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:54,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:54,208 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:54,208 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:54,208 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:54,208 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:54,208 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:54,208 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:54,209 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:54,209 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:54,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:54,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:54,210 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:54,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:54,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:54,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:54,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:54,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:54,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:54,210 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:54,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:54,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:54,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:54,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:54,212 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:54,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:54,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:54,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:54,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:54,214 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:54,214 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:54,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:54,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:54,215 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:54,215 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:54,215 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:54,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:54,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:54,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:54,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:54,215 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:54,215 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:54,215 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:54,215 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:54,215 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:54,215 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:54,215 DEBUG [Thread-1666] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:54,215 DEBUG [Thread-1666] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:54,238 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:54,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40794 deadline: 1680281754237, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:54,242 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:54,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40748 deadline: 1680281754239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:54,242 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:54,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40716 deadline: 1680281754240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:54,243 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:54,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40768 deadline: 1680281754242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:54,316 DEBUG [Thread-1666] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:54,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:54,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:54,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:54,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:54,319 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:54,319 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:54,319 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:54,319 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:54,320 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:54,320 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:54,321 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:54,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:54,321 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:54,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:54,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:54,322 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:54,322 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:54,322 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:54,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:54,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:54,323 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:54,323 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:54,323 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:54,323 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:54,323 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:54,323 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:54,323 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:54,323 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:54,323 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:54,323 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. started... 2023-03-31 16:54:54,323 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. as already flushing 2023-03-31 16:54:54,323 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:54,323 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:54,323 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:54,323 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:54,324 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:54,324 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:54,326 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:54,326 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:54,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:54,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:54,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:54,326 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:54,326 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:54,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:54,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:54,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:54,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:54,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:54,328 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:54,328 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@468f0e11[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:54,328 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:54,328 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:54,329 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:54,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:54,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:54,329 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:54,329 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:54,329 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:54,329 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:54,329 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:54,333 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:54,333 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:54,333 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:54,333 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:54,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:54,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:54,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:54,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:54,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:54,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:54,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:54,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:54,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:54,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:54,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:54,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:54,336 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:54,336 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:54,336 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:54,336 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:54,336 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:54,339 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:54,339 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:54,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:54,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:54,339 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:54,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:54,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:54,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:54,340 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:54,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:54,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:54,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:54,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:54,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:54,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:54,341 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:54,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:54,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:54,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:54,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:54,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:54,343 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:54,343 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:54,343 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:54,343 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:54,343 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:54,343 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:54,343 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:54,343 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:54,343 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:54,343 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:54,344 DEBUG [Thread-1666] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:54,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:54,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:54,344 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:54,343 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:54,344 DEBUG [Thread-1666] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:54,344 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:54,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:54,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:54,444 DEBUG [Thread-1666] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:54,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:54,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:54,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:54,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:54,447 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:54,447 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:54,447 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:54,447 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:54,448 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:54,448 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:54,449 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:54,449 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:54,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:54,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:54,449 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:54,449 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:54,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:54,450 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:54,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:54,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:54,450 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:54,450 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:54,450 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:54,450 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:54,450 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:54,450 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:54,451 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:54,451 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:54,451 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:54,451 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. started... 2023-03-31 16:54:54,451 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. as already flushing 2023-03-31 16:54:54,451 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:54,451 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:54,451 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:54,451 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:54,451 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:54,451 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:54,452 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:54,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:54,452 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:54,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:54,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:54,453 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:54,453 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:54,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:54,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:54,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:54,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:54,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:54,454 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:54,454 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5fb44db2[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:54,454 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:54,454 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:54,457 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:54,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:54,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:54,457 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:54,457 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:54,457 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:54,457 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:54,457 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:54,459 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:54,459 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:54,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:54,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:54,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:54,459 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:54,459 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:54,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:54,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:54,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:54,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:54,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:54,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:54,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:54,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:54,462 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:54,462 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:54,462 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:54,462 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:54,462 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:54,462 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:54,463 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:54,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:54,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:54,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:54,464 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:54,464 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:54,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:54,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:54,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:54,464 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:54,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:54,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:54,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:54,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:54,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:54,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:54,466 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:54,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:54,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:54,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:54,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:54,467 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:54,467 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:54,467 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:54,467 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:54,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:54,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:54,468 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:54,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:54,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:54,468 DEBUG [Thread-1666] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:54,468 DEBUG [Thread-1666] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:54,467 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:54,468 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:54,468 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:54,468 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:54,468 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:54,468 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:54,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:54,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:54,468 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=844 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/C/7d142958f1524c54b91d28c5b07e27ad 2023-03-31 16:54:54,473 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/A/2c2aee8a7bdf467789516978cd0be6f7 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/2c2aee8a7bdf467789516978cd0be6f7 2023-03-31 16:54:54,477 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/2c2aee8a7bdf467789516978cd0be6f7, entries=200, sequenceid=844, filesize=14.3 K 2023-03-31 16:54:54,478 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/B/ca44fadb4f7d4ee0b171ca6a1d9c8035 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/ca44fadb4f7d4ee0b171ca6a1d9c8035 2023-03-31 16:54:54,481 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/ca44fadb4f7d4ee0b171ca6a1d9c8035, entries=150, sequenceid=844, filesize=11.9 K 2023-03-31 16:54:54,482 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/C/7d142958f1524c54b91d28c5b07e27ad as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/7d142958f1524c54b91d28c5b07e27ad 2023-03-31 16:54:54,486 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/7d142958f1524c54b91d28c5b07e27ad, entries=150, sequenceid=844, filesize=11.9 K 2023-03-31 16:54:54,486 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 3ab680f7c89c77c9be754a4581df5bc8 in 868ms, sequenceid=844, compaction requested=true 2023-03-31 16:54:54,487 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3ab680f7c89c77c9be754a4581df5bc8: 2023-03-31 16:54:54,487 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:54:54,487 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 16:54:54,487 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 16:54:54,487 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:54:54,487 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:54:54,488 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39190 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 16:54:54,488 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36750 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 16:54:54,488 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1912): 3ab680f7c89c77c9be754a4581df5bc8/A is initiating minor compaction (all files) 2023-03-31 16:54:54,488 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): 3ab680f7c89c77c9be754a4581df5bc8/B is initiating minor compaction (all files) 2023-03-31 16:54:54,488 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3ab680f7c89c77c9be754a4581df5bc8/A in TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:54,488 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3ab680f7c89c77c9be754a4581df5bc8/B in TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:54,488 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/654233eeecf943a4bb2d01ba2861d573, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/7a38f1c9abf54c96b1180fda896def39, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/2c2aee8a7bdf467789516978cd0be6f7] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp, totalSize=38.3 K 2023-03-31 16:54:54,488 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/12ea4e7eff5d4b728065004a29cfc554, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/e3bfb109cc3e413d865bc65e82158419, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/ca44fadb4f7d4ee0b171ca6a1d9c8035] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp, totalSize=35.9 K 2023-03-31 16:54:54,488 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 654233eeecf943a4bb2d01ba2861d573, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=231, earliestPutTs=1680281692355 2023-03-31 16:54:54,489 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 12ea4e7eff5d4b728065004a29cfc554, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=231, earliestPutTs=1680281692355 2023-03-31 16:54:54,489 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 7a38f1c9abf54c96b1180fda896def39, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=806, earliestPutTs=1680281693490 2023-03-31 16:54:54,489 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting e3bfb109cc3e413d865bc65e82158419, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=806, earliestPutTs=1680281693490 2023-03-31 16:54:54,490 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting ca44fadb4f7d4ee0b171ca6a1d9c8035, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=844, earliestPutTs=1680281693605 2023-03-31 16:54:54,490 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 2c2aee8a7bdf467789516978cd0be6f7, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=844, earliestPutTs=1680281693602 2023-03-31 16:54:54,496 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3ab680f7c89c77c9be754a4581df5bc8#A#compaction#245 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:54:54,496 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): 3ab680f7c89c77c9be754a4581df5bc8#B#compaction#246 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:54:54,509 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/A/fc90c4f61e574f18b3e65b94d6e7ed1d as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/fc90c4f61e574f18b3e65b94d6e7ed1d 2023-03-31 16:54:54,514 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3ab680f7c89c77c9be754a4581df5bc8/A of 3ab680f7c89c77c9be754a4581df5bc8 into fc90c4f61e574f18b3e65b94d6e7ed1d(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:54:54,514 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3ab680f7c89c77c9be754a4581df5bc8: 2023-03-31 16:54:54,514 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8., storeName=3ab680f7c89c77c9be754a4581df5bc8/A, priority=13, startTime=1680281694487; duration=0sec 2023-03-31 16:54:54,514 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:54:54,514 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 16:54:54,515 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36750 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 16:54:54,515 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1912): 3ab680f7c89c77c9be754a4581df5bc8/C is initiating minor compaction (all files) 2023-03-31 16:54:54,515 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3ab680f7c89c77c9be754a4581df5bc8/C in TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:54,515 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/2d04dee6d0f94e379b9e1980afdb654a, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/db96159877b349c989a32da1e10184f6, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/7d142958f1524c54b91d28c5b07e27ad] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp, totalSize=35.9 K 2023-03-31 16:54:54,516 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 2d04dee6d0f94e379b9e1980afdb654a, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=231, earliestPutTs=1680281692355 2023-03-31 16:54:54,516 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting db96159877b349c989a32da1e10184f6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=806, earliestPutTs=1680281693490 2023-03-31 16:54:54,516 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 7d142958f1524c54b91d28c5b07e27ad, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=844, earliestPutTs=1680281693605 2023-03-31 16:54:54,522 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3ab680f7c89c77c9be754a4581df5bc8#C#compaction#247 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:54:54,530 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/C/ca8db8334a614dbbb1b2f33b53104c4d as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/ca8db8334a614dbbb1b2f33b53104c4d 2023-03-31 16:54:54,534 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3ab680f7c89c77c9be754a4581df5bc8/C of 3ab680f7c89c77c9be754a4581df5bc8 into ca8db8334a614dbbb1b2f33b53104c4d(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:54:54,534 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3ab680f7c89c77c9be754a4581df5bc8: 2023-03-31 16:54:54,534 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8., storeName=3ab680f7c89c77c9be754a4581df5bc8/C, priority=13, startTime=1680281694487; duration=0sec 2023-03-31 16:54:54,534 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:54:54,568 DEBUG [Thread-1666] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:54,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:54,570 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:54,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:54,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:54,571 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:54,571 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:54,571 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:54,571 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:54,572 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:54,572 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:54,574 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:54,574 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:54,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:54,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:54,574 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:54,574 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:54,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:54,575 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:54,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:54,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:54,575 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:54,575 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:54,575 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:54,575 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:54,575 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:54,575 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:54,576 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:54,576 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:54,576 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:54,576 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. started... 2023-03-31 16:54:54,576 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3ab680f7c89c77c9be754a4581df5bc8 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-31 16:54:54,576 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=A 2023-03-31 16:54:54,576 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:54,576 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=B 2023-03-31 16:54:54,576 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:54,576 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=C 2023-03-31 16:54:54,576 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:54,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on 3ab680f7c89c77c9be754a4581df5bc8 2023-03-31 16:54:54,690 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. as already flushing 2023-03-31 16:54:54,736 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680281694724, max timestamp=1680281694734Num uniques 859; 2023-03-31 16:54:54,736 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680281694724, max timestamp=1680281694734Num uniques 859; 2023-03-31 16:54:54,736 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680281694724, max timestamp=1680281694734Num uniques 859; 2023-03-31 16:54:54,736 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 16:54:54,736 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 16:54:54,736 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 16:54:54,758 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:54,758 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:54,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 281 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40716 deadline: 1680281754754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:54,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40748 deadline: 1680281754754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:54,759 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:54,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40794 deadline: 1680281754754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:54,760 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680281694752, max timestamp=1680281694758Num uniques 1018; 2023-03-31 16:54:54,760 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680281694752, max timestamp=1680281694758Num uniques 1018; 2023-03-31 16:54:54,760 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-03-31 16:54:54,760 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680281694752, max timestamp=1680281694758Num uniques 1018; 2023-03-31 16:54:54,760 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-03-31 16:54:54,761 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-03-31 16:54:54,787 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:54,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1680281754784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:54,789 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:54,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40768 deadline: 1680281754787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:54,861 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:54,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40794 deadline: 1680281754860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:54,862 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:54,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40748 deadline: 1680281754861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:54,863 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:54,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 283 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40716 deadline: 1680281754861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:54,889 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:54,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1680281754888, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:54,892 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:54,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40768 deadline: 1680281754890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:54,910 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/B/fd064160575342d1a44f215a6e5cd621 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/fd064160575342d1a44f215a6e5cd621 2023-03-31 16:54:54,914 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3ab680f7c89c77c9be754a4581df5bc8/B of 3ab680f7c89c77c9be754a4581df5bc8 into fd064160575342d1a44f215a6e5cd621(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:54:54,914 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3ab680f7c89c77c9be754a4581df5bc8: 2023-03-31 16:54:54,914 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8., storeName=3ab680f7c89c77c9be754a4581df5bc8/B, priority=13, startTime=1680281694487; duration=0sec 2023-03-31 16:54:54,915 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:54:54,987 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=857 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/A/3e4e7da004124a1f8d26fc603c78d4be 2023-03-31 16:54:54,997 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=857 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/B/9dc4c5992230473f9d49755e3bf97dd6 2023-03-31 16:54:55,065 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:55,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40794 deadline: 1680281755063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:55,066 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:55,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40748 deadline: 1680281755064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:55,066 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:55,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 285 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40716 deadline: 1680281755065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:55,092 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:55,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1680281755091, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:55,095 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:55,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40768 deadline: 1680281755093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:55,369 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:55,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40794 deadline: 1680281755367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:55,369 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:55,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40748 deadline: 1680281755368, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:55,369 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:55,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 287 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40716 deadline: 1680281755368, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:55,397 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:55,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1680281755394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:55,397 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:55,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40768 deadline: 1680281755397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:55,406 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=857 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/C/c8594bb4a19448a39ea9b175c21becba 2023-03-31 16:54:55,411 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/A/3e4e7da004124a1f8d26fc603c78d4be as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/3e4e7da004124a1f8d26fc603c78d4be 2023-03-31 16:54:55,416 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/3e4e7da004124a1f8d26fc603c78d4be, entries=150, sequenceid=857, filesize=11.9 K 2023-03-31 16:54:55,417 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/B/9dc4c5992230473f9d49755e3bf97dd6 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/9dc4c5992230473f9d49755e3bf97dd6 2023-03-31 16:54:55,427 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/9dc4c5992230473f9d49755e3bf97dd6, entries=150, sequenceid=857, filesize=11.9 K 2023-03-31 16:54:55,428 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/C/c8594bb4a19448a39ea9b175c21becba as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/c8594bb4a19448a39ea9b175c21becba 2023-03-31 16:54:55,432 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/c8594bb4a19448a39ea9b175c21becba, entries=150, sequenceid=857, filesize=11.9 K 2023-03-31 16:54:55,433 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 3ab680f7c89c77c9be754a4581df5bc8 in 856ms, sequenceid=857, compaction requested=false 2023-03-31 16:54:55,433 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3ab680f7c89c77c9be754a4581df5bc8: 2023-03-31 16:54:55,433 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:55,433 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:55,433 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:55,433 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:55,436 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:55,436 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:55,437 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:55,437 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:55,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:55,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:55,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:55,438 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:55,438 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:55,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:55,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:55,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:55,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:55,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:55,441 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:55,441 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d9a9fc6[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:55,441 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:55,441 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:55,442 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:55,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:55,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:55,442 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:55,442 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:55,442 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:55,443 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:55,443 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:55,444 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:55,444 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:55,444 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:55,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:55,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:55,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:55,444 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:55,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:55,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:55,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:55,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:55,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:55,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:55,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:55,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:55,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:55,448 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:55,448 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:55,448 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:55,448 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:55,448 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:55,450 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:55,450 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:55,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:55,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:55,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:55,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:55,450 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:55,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:55,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:55,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:55,451 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:55,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:55,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:55,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:55,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:55,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:55,452 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:55,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:55,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:55,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:55,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:55,455 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:55,455 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:55,455 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:55,455 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:55,455 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:55,455 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:55,455 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:55,455 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:55,456 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:55,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:55,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:55,456 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:55,456 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:55,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:55,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:55,456 DEBUG [Thread-1666] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:55,456 DEBUG [Thread-1666] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:55,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:55,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:55,556 DEBUG [Thread-1666] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:55,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:55,558 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:55,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:55,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:55,560 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:55,560 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:55,560 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:55,560 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:55,560 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:55,560 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:55,561 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:55,561 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:55,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:55,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:55,561 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:55,562 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:55,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:55,562 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:55,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:55,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:55,562 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:55,563 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:55,563 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:55,563 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:55,563 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:55,563 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:55,563 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:55,563 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:55,563 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:55,563 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. started... 2023-03-31 16:54:55,563 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3ab680f7c89c77c9be754a4581df5bc8 3/3 column families, dataSize=134.18 KB heapSize=384.14 KB 2023-03-31 16:54:55,564 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=A 2023-03-31 16:54:55,564 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:54:55,564 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=B 2023-03-31 16:54:55,564 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:54:55,564 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=C 2023-03-31 16:54:55,564 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:54:55,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on 3ab680f7c89c77c9be754a4581df5bc8 2023-03-31 16:54:55,873 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. as already flushing 2023-03-31 16:54:55,881 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:55,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40748 deadline: 1680281755878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:55,883 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:55,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40794 deadline: 1680281755880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:55,884 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:55,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 292 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40716 deadline: 1680281755881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:55,899 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:55,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1680281755898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:55,903 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:55,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40768 deadline: 1680281755901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:55,978 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=912 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/A/e8b5c731ec8c4e0e8fda7b5011d25c06 2023-03-31 16:54:55,984 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:55,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40748 deadline: 1680281755982, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:55,985 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:55,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40794 deadline: 1680281755984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:55,987 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:55,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 294 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40716 deadline: 1680281755984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:56,187 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:56,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40748 deadline: 1680281756186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:56,188 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:56,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40794 deadline: 1680281756186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:56,190 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:56,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 296 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40716 deadline: 1680281756188, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:56,391 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=912 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/B/296d22848fd14345bf379a7fe7073ab6 2023-03-31 16:54:56,491 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:56,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40748 deadline: 1680281756490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:56,492 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:56,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40794 deadline: 1680281756491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:56,493 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:56,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 298 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40716 deadline: 1680281756491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:56,801 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=912 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/C/5f7fe022293f4aa98ebcb21fff1032dd 2023-03-31 16:54:56,806 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/A/e8b5c731ec8c4e0e8fda7b5011d25c06 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/e8b5c731ec8c4e0e8fda7b5011d25c06 2023-03-31 16:54:56,809 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/e8b5c731ec8c4e0e8fda7b5011d25c06, entries=150, sequenceid=912, filesize=11.9 K 2023-03-31 16:54:56,810 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/B/296d22848fd14345bf379a7fe7073ab6 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/296d22848fd14345bf379a7fe7073ab6 2023-03-31 16:54:56,813 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/296d22848fd14345bf379a7fe7073ab6, entries=150, sequenceid=912, filesize=11.9 K 2023-03-31 16:54:56,814 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/C/5f7fe022293f4aa98ebcb21fff1032dd as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/5f7fe022293f4aa98ebcb21fff1032dd 2023-03-31 16:54:56,817 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/5f7fe022293f4aa98ebcb21fff1032dd, entries=150, sequenceid=912, filesize=11.9 K 2023-03-31 16:54:56,819 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.09 KB/393312, currentSize=46.96 KB/48090 for 3ab680f7c89c77c9be754a4581df5bc8 in 1256ms, sequenceid=912, compaction requested=true 2023-03-31 16:54:56,819 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3ab680f7c89c77c9be754a4581df5bc8: 2023-03-31 16:54:56,819 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:56,819 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:56,819 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:56,819 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:56,819 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:56,819 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:56,821 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:56,821 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:56,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:56,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:56,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:56,821 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:56,821 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:56,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:56,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:56,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:56,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:56,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:56,823 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:56,823 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79e7a89f[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:56,823 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:56,823 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:56,824 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:56,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:56,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:56,824 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:56,824 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:56,824 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:56,825 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:56,825 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:56,826 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:56,826 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:56,826 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:56,826 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:56,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:56,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:56,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:56,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:56,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:56,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:56,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:56,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:56,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:56,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:56,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:56,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:56,828 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:56,828 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:56,828 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:56,828 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:56,828 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:56,834 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:56,834 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:56,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:56,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:56,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:56,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:56,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:56,834 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:56,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:56,834 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:56,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:56,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:56,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:56,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:56,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:56,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:56,835 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:56,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:56,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:56,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:56,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:56,838 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:56,838 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:56,838 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:56,838 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:56,838 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:56,838 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:56,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:56,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:56,838 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:56,838 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:56,838 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:56,838 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:56,838 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:56,838 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:56,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:56,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:56,838 DEBUG [Thread-1666] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:56,838 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:56,838 DEBUG [Thread-1666] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:56,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on 3ab680f7c89c77c9be754a4581df5bc8 2023-03-31 16:54:56,909 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3ab680f7c89c77c9be754a4581df5bc8 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-31 16:54:56,909 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=A 2023-03-31 16:54:56,909 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:56,909 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=B 2023-03-31 16:54:56,909 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:56,909 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=C 2023-03-31 16:54:56,909 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:56,938 DEBUG [Thread-1666] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:56,939 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=109, cellSize=4985, totalHeapSize=11576, min timestamp=1680281696933, max timestamp=1680281696937Num uniques 859; 2023-03-31 16:54:56,939 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=109, cellSize=4985, totalHeapSize=11576, min timestamp=1680281696933, max timestamp=1680281696937Num uniques 859; 2023-03-31 16:54:56,939 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=4985, suffix heap size=117064,new segment heap size=11576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=109 2023-03-31 16:54:56,939 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=4985, suffix heap size=117064,new segment heap size=11576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=109 2023-03-31 16:54:56,939 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=109, cellSize=4985, totalHeapSize=11576, min timestamp=1680281696933, max timestamp=1680281696937Num uniques 859; 2023-03-31 16:54:56,939 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=4985, suffix heap size=117064,new segment heap size=11576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=109 2023-03-31 16:54:56,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:56,941 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:56,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:56,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:56,942 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:56,942 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:56,942 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:56,943 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:56,943 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:56,943 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:56,944 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:56,944 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:56,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:56,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:56,945 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:56,945 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:56,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:56,945 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:56,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:56,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:56,945 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:56,946 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:56,946 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:56,946 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:56,946 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:56,946 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:56,946 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:56,946 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:56,946 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:56,946 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. started... 2023-03-31 16:54:56,946 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. as already flushing 2023-03-31 16:54:56,946 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:56,946 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:56,946 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:56,946 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:56,947 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:56,947 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:56,948 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:56,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:56,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:56,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:56,948 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:56,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:56,948 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:56,948 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:56,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:56,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:56,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:56,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:56,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:56,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4726de34[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:56,950 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:56,950 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:56,952 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:56,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:56,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:56,952 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:56,952 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:56,952 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:56,952 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:56,952 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:56,953 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:56,953 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:56,953 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:56,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:56,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:56,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:56,953 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:56,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:56,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:56,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:56,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:56,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:56,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:56,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:56,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:56,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:56,956 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:56,956 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:56,956 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:56,956 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:56,956 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:56,958 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:56,958 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:56,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:56,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:56,958 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:56,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:56,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:56,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:56,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:56,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:56,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:56,959 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:56,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:56,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:56,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:56,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:56,961 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:56,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:56,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:56,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:56,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:56,962 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:56,963 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:56,963 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:56,963 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:56,963 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:56,963 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:56,963 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:56,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:56,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:56,963 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:56,963 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:56,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:56,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:56,963 DEBUG [Thread-1666] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:56,963 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:56,963 DEBUG [Thread-1666] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:56,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:56,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:56,963 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:56,972 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:56,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 298 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1680281756971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:56,974 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:56,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40768 deadline: 1680281756971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:56,994 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:56,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40748 deadline: 1680281756993, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:56,996 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:56,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 300 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40716 deadline: 1680281756995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:56,998 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:56,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40794 deadline: 1680281756997, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:57,063 DEBUG [Thread-1666] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:57,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:57,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:57,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:57,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:57,067 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:57,067 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:57,067 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:57,067 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:57,067 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:57,067 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:57,070 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:57,070 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:57,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:57,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:57,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:57,070 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:57,070 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:57,071 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:57,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:57,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:57,071 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:57,071 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:57,071 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:57,071 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:57,071 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:57,071 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:57,072 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:57,072 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:57,072 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:57,072 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. started... 2023-03-31 16:54:57,072 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. as already flushing 2023-03-31 16:54:57,072 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:57,072 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:57,072 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:57,072 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:57,072 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:57,072 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:57,073 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:57,073 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:57,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:57,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:57,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:57,074 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:57,074 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:57,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:57,074 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:57,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 300 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1680281757073, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:57,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:57,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:57,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:57,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:57,075 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:57,075 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a02d2bf[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:57,075 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:57,075 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:57,076 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:57,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:57,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:57,076 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:57,076 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:57,076 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:57,077 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:57,077 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:57,077 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:57,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40768 deadline: 1680281757075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:57,078 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:57,078 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:57,078 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:57,078 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:57,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:57,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:57,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:57,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:57,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:57,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:57,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:57,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:57,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:57,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:57,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:57,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:57,080 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:57,080 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:57,080 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:57,080 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:57,080 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:57,083 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:57,083 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:57,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:57,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:57,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:57,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:57,083 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:57,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:57,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:57,083 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:57,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:57,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:57,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:57,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:57,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:57,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:57,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:57,084 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:57,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:57,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:57,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:57,086 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:57,086 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:57,086 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:57,086 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:57,086 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:57,086 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:57,086 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:57,086 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:57,086 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:57,086 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:57,086 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:57,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:57,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:57,086 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:57,086 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:57,087 DEBUG [Thread-1666] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:57,087 DEBUG [Thread-1666] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:57,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:57,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:57,187 DEBUG [Thread-1666] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:57,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:57,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:57,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:57,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:57,191 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:57,191 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:57,191 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:57,191 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:57,191 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:57,192 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:57,193 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:57,193 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:57,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:57,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:57,193 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:57,193 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:57,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:57,194 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:57,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:57,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:57,194 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:57,194 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:57,194 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:57,195 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:57,195 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:57,195 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:57,195 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:57,195 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:57,195 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:57,195 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. started... 2023-03-31 16:54:57,195 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. as already flushing 2023-03-31 16:54:57,195 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:57,195 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:57,195 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:57,195 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:57,195 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:57,195 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:57,197 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:57,197 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:57,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:57,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:57,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:57,198 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:57,198 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:57,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:57,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:57,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:57,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:57,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:57,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:57,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@439f0e96[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:57,200 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:57,200 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:57,202 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:57,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:57,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:57,202 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:57,202 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:57,202 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:57,202 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:57,202 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:57,204 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:57,204 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:57,204 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:57,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:57,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:57,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:57,204 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:57,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:57,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:57,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:57,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:57,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:57,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:57,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:57,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:57,206 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:57,206 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:57,206 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:57,206 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:57,206 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:57,206 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:57,208 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:57,208 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:57,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:57,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:57,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:57,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:57,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:57,208 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:57,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:57,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:57,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:57,208 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:57,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:57,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:57,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:57,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:57,210 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:57,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:57,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:57,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:57,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:57,215 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:57,215 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:57,215 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:57,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:57,215 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:57,215 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:57,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:57,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:57,215 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:57,215 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:57,215 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:57,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:57,216 DEBUG [Thread-1666] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:57,216 DEBUG [Thread-1666] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:57,216 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:57,216 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:57,216 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:57,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:57,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:57,276 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:57,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 302 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1680281757275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:57,281 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:57,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40768 deadline: 1680281757279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:57,316 DEBUG [Thread-1666] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:57,316 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=923 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/A/045570ed7d1c4fb9b03789a80f2ceb12 2023-03-31 16:54:57,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:57,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:57,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:57,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:57,320 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:57,320 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:57,320 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:57,320 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:57,321 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:57,321 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:57,324 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:57,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:57,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:57,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:57,325 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:57,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:57,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:57,325 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:57,326 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:57,326 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:57,326 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:57,327 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:57,327 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:57,327 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:57,327 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:57,327 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:57,327 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:57,327 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:57,327 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:57,327 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. started... 2023-03-31 16:54:57,327 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. as already flushing 2023-03-31 16:54:57,327 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:57,327 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:57,328 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:57,328 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:57,328 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:57,328 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:57,329 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:57,329 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:57,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:57,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:57,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:57,330 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:57,330 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:57,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:57,330 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=923 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/B/00bfb127153c44688c4948bba3ee7ba7 2023-03-31 16:54:57,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:57,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:57,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:57,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:57,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:57,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@778f52a[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:57,333 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:57,333 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:57,334 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:57,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:57,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:57,335 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:57,335 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:57,335 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:57,335 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:57,336 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:57,338 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:57,338 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:57,338 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:57,338 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:57,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:57,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:57,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:57,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:57,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:57,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:57,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:57,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:57,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:57,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:57,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:57,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:57,342 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:57,342 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=923 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/C/0e0b7ffc064d448f8aa3adb0c2f5392a 2023-03-31 16:54:57,342 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:57,342 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:57,342 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:57,342 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:57,344 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:57,344 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:57,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:57,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:57,344 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:57,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:57,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:57,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:57,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:57,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:57,345 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:57,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:57,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:57,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:57,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:57,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:57,347 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/A/045570ed7d1c4fb9b03789a80f2ceb12 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/045570ed7d1c4fb9b03789a80f2ceb12 2023-03-31 16:54:57,347 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:57,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:57,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:57,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:57,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:57,349 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:57,349 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:57,349 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:57,349 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:57,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:57,349 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:57,349 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:57,349 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:57,350 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:57,349 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:57,350 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:57,350 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:57,350 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:57,350 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:57,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:57,350 DEBUG [Thread-1666] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:57,350 DEBUG [Thread-1666] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:57,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:57,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:57,351 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/045570ed7d1c4fb9b03789a80f2ceb12, entries=100, sequenceid=923, filesize=9.5 K 2023-03-31 16:54:57,352 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/B/00bfb127153c44688c4948bba3ee7ba7 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/00bfb127153c44688c4948bba3ee7ba7 2023-03-31 16:54:57,357 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/00bfb127153c44688c4948bba3ee7ba7, entries=100, sequenceid=923, filesize=9.5 K 2023-03-31 16:54:57,358 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/C/0e0b7ffc064d448f8aa3adb0c2f5392a as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/0e0b7ffc064d448f8aa3adb0c2f5392a 2023-03-31 16:54:57,362 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/0e0b7ffc064d448f8aa3adb0c2f5392a, entries=100, sequenceid=923, filesize=9.5 K 2023-03-31 16:54:57,363 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 3ab680f7c89c77c9be754a4581df5bc8 in 455ms, sequenceid=923, compaction requested=true 2023-03-31 16:54:57,363 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3ab680f7c89c77c9be754a4581df5bc8: 2023-03-31 16:54:57,363 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:54:57,363 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 16:54:57,363 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:54:57,363 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:54:57,363 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 16:54:57,364 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46744 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 16:54:57,364 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46744 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 16:54:57,364 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1912): 3ab680f7c89c77c9be754a4581df5bc8/A is initiating minor compaction (all files) 2023-03-31 16:54:57,364 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): 3ab680f7c89c77c9be754a4581df5bc8/B is initiating minor compaction (all files) 2023-03-31 16:54:57,365 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3ab680f7c89c77c9be754a4581df5bc8/A in TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:57,365 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3ab680f7c89c77c9be754a4581df5bc8/B in TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:57,365 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/fc90c4f61e574f18b3e65b94d6e7ed1d, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/3e4e7da004124a1f8d26fc603c78d4be, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/e8b5c731ec8c4e0e8fda7b5011d25c06, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/045570ed7d1c4fb9b03789a80f2ceb12] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp, totalSize=45.6 K 2023-03-31 16:54:57,365 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/fd064160575342d1a44f215a6e5cd621, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/9dc4c5992230473f9d49755e3bf97dd6, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/296d22848fd14345bf379a7fe7073ab6, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/00bfb127153c44688c4948bba3ee7ba7] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp, totalSize=45.6 K 2023-03-31 16:54:57,365 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting fc90c4f61e574f18b3e65b94d6e7ed1d, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=844, earliestPutTs=1680281693605 2023-03-31 16:54:57,365 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting fd064160575342d1a44f215a6e5cd621, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=844, earliestPutTs=1680281693605 2023-03-31 16:54:57,365 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 3e4e7da004124a1f8d26fc603c78d4be, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=857, earliestPutTs=1680281693623 2023-03-31 16:54:57,365 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 9dc4c5992230473f9d49755e3bf97dd6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=857, earliestPutTs=1680281693623 2023-03-31 16:54:57,366 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting e8b5c731ec8c4e0e8fda7b5011d25c06, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=912, earliestPutTs=1680281694782 2023-03-31 16:54:57,366 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 296d22848fd14345bf379a7fe7073ab6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=912, earliestPutTs=1680281694782 2023-03-31 16:54:57,366 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 045570ed7d1c4fb9b03789a80f2ceb12, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=923, earliestPutTs=1680281695879 2023-03-31 16:54:57,366 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 00bfb127153c44688c4948bba3ee7ba7, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=923, earliestPutTs=1680281695879 2023-03-31 16:54:57,374 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3ab680f7c89c77c9be754a4581df5bc8#A#compaction#258 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:54:57,375 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): 3ab680f7c89c77c9be754a4581df5bc8#B#compaction#257 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:54:57,402 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/A/09ebd0ffb0944b459a2f9da6ff0bdd11 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/09ebd0ffb0944b459a2f9da6ff0bdd11 2023-03-31 16:54:57,407 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3ab680f7c89c77c9be754a4581df5bc8/A of 3ab680f7c89c77c9be754a4581df5bc8 into 09ebd0ffb0944b459a2f9da6ff0bdd11(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:54:57,407 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3ab680f7c89c77c9be754a4581df5bc8: 2023-03-31 16:54:57,407 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8., storeName=3ab680f7c89c77c9be754a4581df5bc8/A, priority=12, startTime=1680281697363; duration=0sec 2023-03-31 16:54:57,407 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:54:57,407 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 16:54:57,408 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46744 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 16:54:57,408 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1912): 3ab680f7c89c77c9be754a4581df5bc8/C is initiating minor compaction (all files) 2023-03-31 16:54:57,408 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3ab680f7c89c77c9be754a4581df5bc8/C in TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:57,409 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/ca8db8334a614dbbb1b2f33b53104c4d, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/c8594bb4a19448a39ea9b175c21becba, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/5f7fe022293f4aa98ebcb21fff1032dd, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/0e0b7ffc064d448f8aa3adb0c2f5392a] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp, totalSize=45.6 K 2023-03-31 16:54:57,409 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting ca8db8334a614dbbb1b2f33b53104c4d, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=844, earliestPutTs=1680281693605 2023-03-31 16:54:57,409 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting c8594bb4a19448a39ea9b175c21becba, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=857, earliestPutTs=1680281693623 2023-03-31 16:54:57,409 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 5f7fe022293f4aa98ebcb21fff1032dd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=912, earliestPutTs=1680281694782 2023-03-31 16:54:57,410 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 0e0b7ffc064d448f8aa3adb0c2f5392a, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=923, earliestPutTs=1680281695879 2023-03-31 16:54:57,418 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3ab680f7c89c77c9be754a4581df5bc8#C#compaction#259 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:54:57,430 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/C/e4d45c15f55d47e4b719b39f9360daa0 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/e4d45c15f55d47e4b719b39f9360daa0 2023-03-31 16:54:57,435 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3ab680f7c89c77c9be754a4581df5bc8/C of 3ab680f7c89c77c9be754a4581df5bc8 into e4d45c15f55d47e4b719b39f9360daa0(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:54:57,435 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3ab680f7c89c77c9be754a4581df5bc8: 2023-03-31 16:54:57,435 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8., storeName=3ab680f7c89c77c9be754a4581df5bc8/C, priority=12, startTime=1680281697363; duration=0sec 2023-03-31 16:54:57,435 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:54:57,450 DEBUG [Thread-1666] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:57,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:57,452 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:57,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:57,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:57,453 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:57,453 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:57,453 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:57,453 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:57,453 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:57,453 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:57,454 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:57,454 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:57,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:57,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:57,455 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:57,455 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:57,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:57,455 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:57,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:57,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:57,456 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:57,456 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:57,456 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:57,456 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:57,456 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:57,456 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:57,456 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:57,456 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:57,456 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:57,456 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. started... 2023-03-31 16:54:57,457 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3ab680f7c89c77c9be754a4581df5bc8 3/3 column families, dataSize=127.47 KB heapSize=369.75 KB 2023-03-31 16:54:57,457 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=A 2023-03-31 16:54:57,457 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:54:57,457 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=B 2023-03-31 16:54:57,457 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:54:57,457 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=C 2023-03-31 16:54:57,457 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:54:57,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on 3ab680f7c89c77c9be754a4581df5bc8 2023-03-31 16:54:57,578 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. as already flushing 2023-03-31 16:54:57,593 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:57,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 309 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1680281757591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:57,594 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:57,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40768 deadline: 1680281757591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:57,695 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:57,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40768 deadline: 1680281757694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:57,695 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:57,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 311 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1680281757695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:57,797 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/B/ab7aea1077564a9099510a0b1276f101 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/ab7aea1077564a9099510a0b1276f101 2023-03-31 16:54:57,801 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3ab680f7c89c77c9be754a4581df5bc8/B of 3ab680f7c89c77c9be754a4581df5bc8 into ab7aea1077564a9099510a0b1276f101(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:54:57,801 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3ab680f7c89c77c9be754a4581df5bc8: 2023-03-31 16:54:57,801 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8., storeName=3ab680f7c89c77c9be754a4581df5bc8/B, priority=12, startTime=1680281697363; duration=0sec 2023-03-31 16:54:57,801 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:54:57,866 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=962 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/A/836b7b6f46db48a99ba766ca44b546d7 2023-03-31 16:54:57,899 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:57,899 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:57,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40768 deadline: 1680281757897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:57,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 313 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1680281757897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:58,004 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:58,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40748 deadline: 1680281758003, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:58,008 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:58,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 302 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40716 deadline: 1680281758006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:58,008 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:58,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40794 deadline: 1680281758007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:58,201 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:58,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 315 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1680281758201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:58,203 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:58,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40768 deadline: 1680281758202, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:58,276 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=962 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/B/aab07af971d84b3a97dc55508856235e 2023-03-31 16:54:58,686 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=962 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/C/9d517bbd93fc4e7797cfb99815a88baf 2023-03-31 16:54:58,690 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/A/836b7b6f46db48a99ba766ca44b546d7 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/836b7b6f46db48a99ba766ca44b546d7 2023-03-31 16:54:58,694 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/836b7b6f46db48a99ba766ca44b546d7, entries=150, sequenceid=962, filesize=11.9 K 2023-03-31 16:54:58,695 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/B/aab07af971d84b3a97dc55508856235e as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/aab07af971d84b3a97dc55508856235e 2023-03-31 16:54:58,698 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/aab07af971d84b3a97dc55508856235e, entries=150, sequenceid=962, filesize=11.9 K 2023-03-31 16:54:58,699 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/C/9d517bbd93fc4e7797cfb99815a88baf as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/9d517bbd93fc4e7797cfb99815a88baf 2023-03-31 16:54:58,702 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/9d517bbd93fc4e7797cfb99815a88baf, entries=150, sequenceid=962, filesize=11.9 K 2023-03-31 16:54:58,703 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~369.70 KB/378576, currentSize=53.67 KB/54960 for 3ab680f7c89c77c9be754a4581df5bc8 in 1247ms, sequenceid=962, compaction requested=false 2023-03-31 16:54:58,703 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3ab680f7c89c77c9be754a4581df5bc8: 2023-03-31 16:54:58,703 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:58,703 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:58,703 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:58,703 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:58,703 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:58,703 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:58,705 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:58,705 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:58,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:58,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:58,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:58,705 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:58,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:58,705 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:58,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on 3ab680f7c89c77c9be754a4581df5bc8 2023-03-31 16:54:58,705 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3ab680f7c89c77c9be754a4581df5bc8 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-31 16:54:58,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:58,706 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=A 2023-03-31 16:54:58,706 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:58,706 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=B 2023-03-31 16:54:58,706 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:58,706 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=C 2023-03-31 16:54:58,706 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:58,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:58,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:58,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:58,707 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:58,707 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76de5c8f[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:58,707 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:58,707 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:58,709 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:58,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:58,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:58,709 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:58,709 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:58,709 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:58,709 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:58,709 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:58,711 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:58,711 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:58,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:58,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:58,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:58,711 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:58,712 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:58,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:58,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:58,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:58,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:58,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:58,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:58,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:58,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:58,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:58,714 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:58,714 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:58,714 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:58,715 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:58,715 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:58,716 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=975 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/A/beae9347000340bda38546557d5fc966 2023-03-31 16:54:58,716 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:58,716 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:58,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:58,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:58,716 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:58,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:58,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:58,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:58,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:58,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:58,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:58,717 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:58,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:58,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:58,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:58,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:58,718 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:58,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:58,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:58,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:58,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:58,721 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:58,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:58,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:58,721 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:58,721 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:58,721 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:58,721 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:58,721 DEBUG [Thread-1666] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:58,721 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:58,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:58,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:58,721 DEBUG [Thread-1666] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:58,721 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:58,721 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:58,722 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:58,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:58,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:58,722 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:58,722 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:58,734 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:58,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40768 deadline: 1680281758732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:58,735 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:58,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 328 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1680281758733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:58,822 DEBUG [Thread-1666] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:58,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:58,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:58,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:58,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:58,825 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:58,825 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:58,825 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:58,825 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:58,825 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:58,825 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:58,826 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:58,826 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:58,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:58,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:58,827 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:58,827 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:58,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:58,827 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:58,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:58,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:58,827 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:58,827 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:58,828 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:58,828 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:58,828 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:58,828 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:58,828 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:58,828 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:58,828 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:58,828 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. started... 2023-03-31 16:54:58,828 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. as already flushing 2023-03-31 16:54:58,828 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:58,828 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:58,828 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:58,828 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:58,828 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:58,828 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:58,831 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:58,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:58,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:58,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:58,831 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:58,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:58,831 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:58,832 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:58,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:58,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:58,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:58,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:58,833 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:58,833 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:58,833 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ae5efc1[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:58,833 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:58,834 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:58,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:58,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:58,834 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:58,834 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:58,834 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:58,834 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:58,834 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:58,835 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:58,835 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:58,835 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:58,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:58,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:58,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:58,835 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:58,836 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:58,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40768 deadline: 1680281758836, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:58,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:58,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:58,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:58,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:58,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:58,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:58,837 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:58,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:58,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 330 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1680281758836, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:58,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:58,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:58,838 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:58,838 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:58,838 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:58,838 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:58,838 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:58,839 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:58,839 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:58,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:58,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:58,840 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:58,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:58,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:58,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:58,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:58,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:58,840 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:58,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:58,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:58,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:58,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:58,841 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:58,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:58,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:58,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:58,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:58,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:58,844 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:58,844 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:58,844 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:58,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:58,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:58,845 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:58,844 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:58,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:58,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:58,845 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:58,845 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:58,845 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:58,845 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:58,845 DEBUG [Thread-1666] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:58,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:58,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:58,845 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:58,845 DEBUG [Thread-1666] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:58,845 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:58,945 DEBUG [Thread-1666] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:58,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:58,947 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:58,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:58,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:58,949 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:58,949 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:58,949 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:58,949 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:58,949 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:58,949 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:58,953 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:58,953 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:58,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:58,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:58,953 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:58,953 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:58,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:58,954 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:58,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:58,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:58,954 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:58,954 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:58,954 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:58,955 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:58,955 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:58,955 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:58,955 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:58,955 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:58,955 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:58,955 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. started... 2023-03-31 16:54:58,955 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. as already flushing 2023-03-31 16:54:58,955 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:58,955 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:58,955 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:58,955 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:58,955 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:58,955 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:58,957 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:58,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:58,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:58,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:58,957 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:58,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:58,957 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:58,957 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:58,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:58,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:58,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:58,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:58,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:58,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61b5a1b5[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:58,958 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:58,958 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:58,960 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:58,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:58,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:58,960 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:58,960 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:58,961 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:58,960 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:58,961 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:58,962 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:58,962 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:58,962 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:58,962 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:58,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:58,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:58,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:58,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:58,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:58,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:58,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:58,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:58,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:58,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:58,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:58,964 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:58,964 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:58,964 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:58,964 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:58,964 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:58,964 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:58,966 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:58,966 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:58,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:58,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:58,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:58,966 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:58,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:58,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:58,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:58,966 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:58,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:58,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:58,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:58,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:58,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:58,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:58,968 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:58,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:58,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:58,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:58,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:58,969 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:58,969 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:58,970 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:58,970 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:58,970 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:58,970 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:58,970 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:58,970 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:58,970 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:58,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:58,970 DEBUG [Thread-1666] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:58,970 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:58,970 DEBUG [Thread-1666] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:58,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:58,970 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:58,970 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:58,970 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:58,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:58,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:59,039 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:59,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40768 deadline: 1680281759038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,040 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:59,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 332 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1680281759040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,070 DEBUG [Thread-1666] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:59,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:59,072 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:59,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:59,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:59,073 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:59,073 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:59,074 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:59,074 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:59,074 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:59,074 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:59,075 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,075 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:59,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:59,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:59,075 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,075 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:59,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:59,076 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:59,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:59,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:59,076 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:59,076 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:59,077 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:59,077 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:59,077 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:59,077 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:59,077 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:59,077 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:59,077 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:59,077 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. started... 2023-03-31 16:54:59,077 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. as already flushing 2023-03-31 16:54:59,077 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:59,077 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:59,077 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:59,077 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,077 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:59,077 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:59,080 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,080 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:59,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:59,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:59,080 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:59,080 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:59,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:59,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:59,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:59,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:59,081 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:59,081 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:59,081 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:59,081 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21a4d90c[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:59,082 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:59,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:59,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:59,083 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:59,083 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,083 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:59,083 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:59,083 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:59,084 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,084 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:59,084 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:59,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:59,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:59,084 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:59,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:59,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:59,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:59,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:59,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:59,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:59,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:59,086 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:59,086 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:59,086 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:59,086 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:59,086 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:59,088 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:59,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:59,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:59,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:59,088 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:59,088 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:59,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:59,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:59,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:59,088 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:59,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:59,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:59,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:59,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:59,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:59,089 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:59,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:59,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,093 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,093 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:59,093 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:59,093 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,093 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:59,093 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:59,093 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:59,093 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:59,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:59,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:59,093 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:59,093 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:59,093 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:59,093 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:59,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:59,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:59,093 DEBUG [Thread-1666] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:59,093 DEBUG [Thread-1666] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:59,093 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:59,130 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=975 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/B/6f5a347face5438c99636caf356655d8 2023-03-31 16:54:59,140 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=975 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/C/7d970e00555e4f21b6013042361665fd 2023-03-31 16:54:59,144 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/A/beae9347000340bda38546557d5fc966 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/beae9347000340bda38546557d5fc966 2023-03-31 16:54:59,148 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/beae9347000340bda38546557d5fc966, entries=150, sequenceid=975, filesize=11.9 K 2023-03-31 16:54:59,149 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/B/6f5a347face5438c99636caf356655d8 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/6f5a347face5438c99636caf356655d8 2023-03-31 16:54:59,153 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/6f5a347face5438c99636caf356655d8, entries=100, sequenceid=975, filesize=9.5 K 2023-03-31 16:54:59,153 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/C/7d970e00555e4f21b6013042361665fd as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/7d970e00555e4f21b6013042361665fd 2023-03-31 16:54:59,157 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/7d970e00555e4f21b6013042361665fd, entries=100, sequenceid=975, filesize=9.5 K 2023-03-31 16:54:59,158 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 3ab680f7c89c77c9be754a4581df5bc8 in 453ms, sequenceid=975, compaction requested=true 2023-03-31 16:54:59,158 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3ab680f7c89c77c9be754a4581df5bc8: 2023-03-31 16:54:59,158 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:54:59,158 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:54:59,158 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 16:54:59,158 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 16:54:59,158 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:54:59,159 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37134 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 16:54:59,159 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1912): 3ab680f7c89c77c9be754a4581df5bc8/A is initiating minor compaction (all files) 2023-03-31 16:54:59,159 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34694 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 16:54:59,159 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3ab680f7c89c77c9be754a4581df5bc8/A in TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:59,159 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): 3ab680f7c89c77c9be754a4581df5bc8/B is initiating minor compaction (all files) 2023-03-31 16:54:59,159 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/09ebd0ffb0944b459a2f9da6ff0bdd11, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/836b7b6f46db48a99ba766ca44b546d7, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/beae9347000340bda38546557d5fc966] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp, totalSize=36.3 K 2023-03-31 16:54:59,159 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3ab680f7c89c77c9be754a4581df5bc8/B in TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:59,159 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/ab7aea1077564a9099510a0b1276f101, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/aab07af971d84b3a97dc55508856235e, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/6f5a347face5438c99636caf356655d8] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp, totalSize=33.9 K 2023-03-31 16:54:59,160 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 09ebd0ffb0944b459a2f9da6ff0bdd11, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=923, earliestPutTs=1680281694783 2023-03-31 16:54:59,160 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting ab7aea1077564a9099510a0b1276f101, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=923, earliestPutTs=1680281694783 2023-03-31 16:54:59,160 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 836b7b6f46db48a99ba766ca44b546d7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=962, earliestPutTs=1680281696951 2023-03-31 16:54:59,160 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting aab07af971d84b3a97dc55508856235e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=962, earliestPutTs=1680281696951 2023-03-31 16:54:59,160 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting beae9347000340bda38546557d5fc966, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=975, earliestPutTs=1680281697590 2023-03-31 16:54:59,160 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 6f5a347face5438c99636caf356655d8, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=975, earliestPutTs=1680281697590 2023-03-31 16:54:59,171 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3ab680f7c89c77c9be754a4581df5bc8#A#compaction#266 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:54:59,177 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): 3ab680f7c89c77c9be754a4581df5bc8#B#compaction#267 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:54:59,193 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/A/34d9013ab93f484b96736f66a0b54e61 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/34d9013ab93f484b96736f66a0b54e61 2023-03-31 16:54:59,194 DEBUG [Thread-1666] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:59,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:59,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:59,199 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3ab680f7c89c77c9be754a4581df5bc8/A of 3ab680f7c89c77c9be754a4581df5bc8 into 34d9013ab93f484b96736f66a0b54e61(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:54:59,199 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3ab680f7c89c77c9be754a4581df5bc8: 2023-03-31 16:54:59,199 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8., storeName=3ab680f7c89c77c9be754a4581df5bc8/A, priority=13, startTime=1680281699158; duration=0sec 2023-03-31 16:54:59,199 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:54:59,199 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 16:54:59,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:59,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:59,200 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:59,200 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:59,200 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:59,200 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:59,200 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34694 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 16:54:59,201 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1912): 3ab680f7c89c77c9be754a4581df5bc8/C is initiating minor compaction (all files) 2023-03-31 16:54:59,201 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3ab680f7c89c77c9be754a4581df5bc8/C in TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:59,201 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/e4d45c15f55d47e4b719b39f9360daa0, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/9d517bbd93fc4e7797cfb99815a88baf, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/7d970e00555e4f21b6013042361665fd] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp, totalSize=33.9 K 2023-03-31 16:54:59,201 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:59,201 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:59,201 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting e4d45c15f55d47e4b719b39f9360daa0, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=923, earliestPutTs=1680281694783 2023-03-31 16:54:59,202 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 9d517bbd93fc4e7797cfb99815a88baf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=962, earliestPutTs=1680281696951 2023-03-31 16:54:59,202 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:59,202 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:59,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:59,202 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 7d970e00555e4f21b6013042361665fd, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=975, earliestPutTs=1680281697590 2023-03-31 16:54:59,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:59,203 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,203 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:59,203 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:59,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:59,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:59,204 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:59,212 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:59,212 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:59,212 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:59,212 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:59,212 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:59,213 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3ab680f7c89c77c9be754a4581df5bc8#C#compaction#268 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:54:59,216 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:59,216 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:59,216 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:59,216 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. started... 2023-03-31 16:54:59,217 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3ab680f7c89c77c9be754a4581df5bc8 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-31 16:54:59,217 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=A 2023-03-31 16:54:59,217 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:59,217 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=B 2023-03-31 16:54:59,217 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:59,217 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=C 2023-03-31 16:54:59,217 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:59,230 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/B/f9f0093211874ef0bde6f36d400a105d as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/f9f0093211874ef0bde6f36d400a105d 2023-03-31 16:54:59,236 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3ab680f7c89c77c9be754a4581df5bc8/B of 3ab680f7c89c77c9be754a4581df5bc8 into f9f0093211874ef0bde6f36d400a105d(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:54:59,236 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3ab680f7c89c77c9be754a4581df5bc8: 2023-03-31 16:54:59,236 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8., storeName=3ab680f7c89c77c9be754a4581df5bc8/B, priority=13, startTime=1680281699158; duration=0sec 2023-03-31 16:54:59,236 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:54:59,274 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=996 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/A/5d0f854716dc4ae18057d463657f320c 2023-03-31 16:54:59,303 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=996 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/B/0e09b30a3f844ca289d0053a1aa51691 2023-03-31 16:54:59,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on 3ab680f7c89c77c9be754a4581df5bc8 2023-03-31 16:54:59,342 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. as already flushing 2023-03-31 16:54:59,345 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=996 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/C/d78c305405d8415aaadb5cf7c4b3d302 2023-03-31 16:54:59,351 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/A/5d0f854716dc4ae18057d463657f320c as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/5d0f854716dc4ae18057d463657f320c 2023-03-31 16:54:59,355 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/5d0f854716dc4ae18057d463657f320c, entries=150, sequenceid=996, filesize=11.9 K 2023-03-31 16:54:59,357 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/B/0e09b30a3f844ca289d0053a1aa51691 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/0e09b30a3f844ca289d0053a1aa51691 2023-03-31 16:54:59,361 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/0e09b30a3f844ca289d0053a1aa51691, entries=150, sequenceid=996, filesize=11.9 K 2023-03-31 16:54:59,362 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:59,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 338 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1680281759359, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,363 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/C/d78c305405d8415aaadb5cf7c4b3d302 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/d78c305405d8415aaadb5cf7c4b3d302 2023-03-31 16:54:59,364 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:59,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40768 deadline: 1680281759361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,368 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/d78c305405d8415aaadb5cf7c4b3d302, entries=150, sequenceid=996, filesize=11.9 K 2023-03-31 16:54:59,369 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 3ab680f7c89c77c9be754a4581df5bc8 in 152ms, sequenceid=996, compaction requested=false 2023-03-31 16:54:59,369 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3ab680f7c89c77c9be754a4581df5bc8: 2023-03-31 16:54:59,369 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:59,369 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:59,369 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:59,369 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,370 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:59,370 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:59,371 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:59,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:59,371 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:59,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:59,372 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:59,372 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:59,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:59,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:59,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:59,374 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:59,375 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@251ad393[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:59,375 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:59,375 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:59,376 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:59,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:59,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:59,376 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:59,376 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:59,376 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:59,399 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,399 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:59,401 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,401 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:59,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:59,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:59,401 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:59,401 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:59,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:59,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:59,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:59,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:59,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:59,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:59,405 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:59,405 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:59,405 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:59,405 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:59,405 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:59,405 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:59,407 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:59,407 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:59,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:59,407 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:59,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:59,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:59,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:59,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:59,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:59,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:59,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:59,409 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:59,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:59,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:59,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:59,411 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:59,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:59,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,413 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:59,413 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:59,413 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,413 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:59,413 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:59,413 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,413 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:59,413 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:59,413 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:59,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:59,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:59,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:59,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:59,413 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:59,414 DEBUG [Thread-1666] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:59,414 DEBUG [Thread-1666] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:59,414 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:59,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:59,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:59,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on 3ab680f7c89c77c9be754a4581df5bc8 2023-03-31 16:54:59,467 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3ab680f7c89c77c9be754a4581df5bc8 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-31 16:54:59,467 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=A 2023-03-31 16:54:59,467 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:59,467 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=B 2023-03-31 16:54:59,467 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:59,467 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=C 2023-03-31 16:54:59,467 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:59,491 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=1010 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/A/1d3136f776ba45b29debf469425c5075 2023-03-31 16:54:59,504 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=1010 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/B/5ddd24c9f83249ceb6b51f9a7b6bfd98 2023-03-31 16:54:59,509 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:59,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 349 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1680281759506, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,512 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:59,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40768 deadline: 1680281759509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,514 DEBUG [Thread-1666] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:59,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:59,515 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=1010 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/C/a54162cd97bb4bb09dfe21b2120d9ee5 2023-03-31 16:54:59,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:59,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:59,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:59,518 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:59,518 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:59,518 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:59,518 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:59,520 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:59,520 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:59,521 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/A/1d3136f776ba45b29debf469425c5075 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/1d3136f776ba45b29debf469425c5075 2023-03-31 16:54:59,521 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:59,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:59,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:59,522 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:59,522 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,522 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:59,523 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:59,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:59,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:59,523 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:59,524 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:59,524 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:59,524 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:59,524 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:59,524 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:59,524 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:59,524 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:59,524 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:59,524 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. started... 2023-03-31 16:54:59,524 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. as already flushing 2023-03-31 16:54:59,524 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:59,525 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:59,525 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:59,525 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,525 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:59,525 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:59,530 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/1d3136f776ba45b29debf469425c5075, entries=150, sequenceid=1010, filesize=11.9 K 2023-03-31 16:54:59,531 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/B/5ddd24c9f83249ceb6b51f9a7b6bfd98 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/5ddd24c9f83249ceb6b51f9a7b6bfd98 2023-03-31 16:54:59,532 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:59,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:59,533 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:59,533 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:59,533 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:59,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:59,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:59,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:59,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:59,535 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:59,535 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5074ab16[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:59,535 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:59,535 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:59,536 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/5ddd24c9f83249ceb6b51f9a7b6bfd98, entries=150, sequenceid=1010, filesize=11.9 K 2023-03-31 16:54:59,537 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:59,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:59,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:59,537 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:59,537 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/C/a54162cd97bb4bb09dfe21b2120d9ee5 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/a54162cd97bb4bb09dfe21b2120d9ee5 2023-03-31 16:54:59,537 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,537 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:59,537 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:59,537 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:59,541 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,541 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:59,541 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:59,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:59,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:59,541 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:59,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:59,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:59,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:59,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:59,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:59,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:59,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:59,553 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:59,553 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:59,553 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:59,553 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:59,553 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:59,555 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:59,555 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:59,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:59,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:59,555 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:59,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:59,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:59,555 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/a54162cd97bb4bb09dfe21b2120d9ee5, entries=150, sequenceid=1010, filesize=11.9 K 2023-03-31 16:54:59,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:59,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:59,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:59,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:59,556 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 3ab680f7c89c77c9be754a4581df5bc8 in 89ms, sequenceid=1010, compaction requested=true 2023-03-31 16:54:59,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:59,556 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3ab680f7c89c77c9be754a4581df5bc8: 2023-03-31 16:54:59,556 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:54:59,556 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:54:59,556 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-31 16:54:59,556 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 16:54:59,556 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:59,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:59,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:59,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,558 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,558 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37240 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 16:54:59,558 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): 3ab680f7c89c77c9be754a4581df5bc8/A is initiating minor compaction (all files) 2023-03-31 16:54:59,558 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3ab680f7c89c77c9be754a4581df5bc8/A in TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:59,558 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/34d9013ab93f484b96736f66a0b54e61, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/5d0f854716dc4ae18057d463657f320c, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/1d3136f776ba45b29debf469425c5075] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp, totalSize=36.4 K 2023-03-31 16:54:59,559 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 34d9013ab93f484b96736f66a0b54e61, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=975, earliestPutTs=1680281696951 2023-03-31 16:54:59,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:59,559 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 5d0f854716dc4ae18057d463657f320c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=996, earliestPutTs=1680281698730 2023-03-31 16:54:59,560 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 1d3136f776ba45b29debf469425c5075, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1010, earliestPutTs=1680281699350 2023-03-31 16:54:59,564 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:59,564 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:59,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:59,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:59,565 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:59,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:59,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:59,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:59,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:59,565 DEBUG [Thread-1666] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:59,565 DEBUG [Thread-1666] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:59,565 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,565 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:59,565 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:59,565 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,565 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:59,565 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:59,565 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:59,565 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:59,573 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): 3ab680f7c89c77c9be754a4581df5bc8#A#compaction#275 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:54:59,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on 3ab680f7c89c77c9be754a4581df5bc8 2023-03-31 16:54:59,613 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3ab680f7c89c77c9be754a4581df5bc8 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-31 16:54:59,614 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=A 2023-03-31 16:54:59,614 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:59,614 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=B 2023-03-31 16:54:59,614 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:59,614 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=C 2023-03-31 16:54:59,614 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:54:59,636 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:59,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 356 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1680281759633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,638 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:59,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40768 deadline: 1680281759636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,648 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/C/8fd40085a0e84570b817f748d032fa94 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/8fd40085a0e84570b817f748d032fa94 2023-03-31 16:54:59,652 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3ab680f7c89c77c9be754a4581df5bc8/C of 3ab680f7c89c77c9be754a4581df5bc8 into 8fd40085a0e84570b817f748d032fa94(size=12.6 K), total size for store is 36.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:54:59,653 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3ab680f7c89c77c9be754a4581df5bc8: 2023-03-31 16:54:59,653 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8., storeName=3ab680f7c89c77c9be754a4581df5bc8/C, priority=13, startTime=1680281699158; duration=0sec 2023-03-31 16:54:59,653 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-31 16:54:59,653 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 16:54:59,654 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37240 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 16:54:59,654 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1912): 3ab680f7c89c77c9be754a4581df5bc8/B is initiating minor compaction (all files) 2023-03-31 16:54:59,654 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3ab680f7c89c77c9be754a4581df5bc8/B in TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:59,654 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/f9f0093211874ef0bde6f36d400a105d, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/0e09b30a3f844ca289d0053a1aa51691, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/5ddd24c9f83249ceb6b51f9a7b6bfd98] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp, totalSize=36.4 K 2023-03-31 16:54:59,654 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting f9f0093211874ef0bde6f36d400a105d, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=975, earliestPutTs=1680281696951 2023-03-31 16:54:59,655 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 0e09b30a3f844ca289d0053a1aa51691, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=996, earliestPutTs=1680281698730 2023-03-31 16:54:59,655 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 5ddd24c9f83249ceb6b51f9a7b6bfd98, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1010, earliestPutTs=1680281699350 2023-03-31 16:54:59,660 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3ab680f7c89c77c9be754a4581df5bc8#B#compaction#277 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:54:59,665 DEBUG [Thread-1666] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:59,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:59,667 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:59,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:59,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:59,668 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:59,668 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:59,668 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:59,668 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:59,669 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:59,669 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:59,670 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:59,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:59,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:59,670 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:59,670 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,670 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:59,671 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:59,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:59,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:59,671 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:59,671 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:59,671 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:59,671 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:59,671 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:59,671 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:59,672 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:59,672 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:59,672 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:59,672 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. started... 2023-03-31 16:54:59,672 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. as already flushing 2023-03-31 16:54:59,672 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:59,672 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:59,672 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:59,672 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,672 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:59,672 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:59,673 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:59,674 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:59,674 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:59,674 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:59,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:59,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:59,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:59,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:59,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:59,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:59,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c0000a4[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:59,676 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:59,676 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:59,677 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:59,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:59,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:59,677 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:59,677 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:59,677 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:59,677 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,677 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:59,679 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,679 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:59,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:59,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:59,679 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:59,680 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:59,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:59,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:59,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:59,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:59,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:59,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:59,682 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:59,682 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:59,682 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:59,682 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:59,682 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:59,682 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:59,683 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:59,683 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:59,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:59,683 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:59,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:59,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:59,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:59,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:59,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:59,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:59,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:59,685 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:59,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:59,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:59,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:59,686 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:59,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:59,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,687 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:59,688 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:59,688 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:59,688 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:59,688 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:59,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:59,688 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:59,688 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:59,688 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:59,688 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:59,688 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,688 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:59,688 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:59,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:59,688 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:59,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:59,688 DEBUG [Thread-1666] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:59,689 DEBUG [Thread-1666] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:59,739 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:59,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 358 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1680281759738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,741 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:59,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40768 deadline: 1680281759740, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,789 DEBUG [Thread-1666] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:59,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:59,791 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:59,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:59,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:59,792 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:59,792 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:59,792 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:59,792 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:59,792 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:59,793 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:59,794 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:59,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:59,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:59,794 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:59,794 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,794 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:59,794 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:59,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:59,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:59,795 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:59,795 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:59,795 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:59,795 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:59,795 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:59,795 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:59,795 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:59,796 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:59,796 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:59,796 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. started... 2023-03-31 16:54:59,796 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. as already flushing 2023-03-31 16:54:59,796 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:59,796 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:59,796 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:59,796 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,796 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:59,796 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:59,798 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:59,798 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:59,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:59,798 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:59,798 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:59,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:59,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:59,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:59,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:59,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:59,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1bebd691[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:59,800 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:59,800 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:59,801 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:59,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:59,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:59,801 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:59,801 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:59,801 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:59,801 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,801 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:59,802 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,802 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:59,802 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:59,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:59,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:59,803 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:59,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:59,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:59,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:59,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:59,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:59,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:59,805 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:59,805 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:59,805 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:59,805 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:59,805 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:59,805 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:59,806 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:59,806 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:59,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:59,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:59,806 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:59,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:59,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:59,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:59,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:59,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:59,807 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:59,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:59,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:59,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:59,808 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:59,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:59,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:59,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,812 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:59,812 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,812 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:59,812 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:59,812 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:59,812 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:59,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:59,812 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:59,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:59,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:59,812 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:59,812 DEBUG [Thread-1666] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:59,812 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:59,813 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:59,813 DEBUG [Thread-1666] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:59,813 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:59,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:59,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:59,913 DEBUG [Thread-1666] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:54:59,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:54:59,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:54:59,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:54:59,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:54:59,916 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:54:59,916 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:59,916 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:54:59,916 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:54:59,916 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:59,916 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:59,917 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:59,917 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:59,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:59,918 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,918 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:54:59,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:59,918 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:59,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:54:59,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:59,919 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:54:59,919 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:54:59,919 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:54:59,919 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:54:59,919 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:54:59,919 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:54:59,919 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:54:59,919 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:54:59,919 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:59,919 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. started... 2023-03-31 16:54:59,919 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. as already flushing 2023-03-31 16:54:59,919 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:54:59,919 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:54:59,920 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:54:59,920 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,920 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:54:59,920 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:54:59,921 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:59,921 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:59,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:59,921 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:59,922 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:54:59,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:59,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:59,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:59,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:59,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:54:59,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@689e0834[Count = 0] remaining members to acquire global barrier 2023-03-31 16:54:59,923 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:54:59,923 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:59,924 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:59,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:59,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:59,924 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:54:59,924 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:54:59,924 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,924 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:54:59,924 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:54:59,931 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,931 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:54:59,931 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:59,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:59,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:59,931 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:54:59,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:59,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:59,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:59,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:59,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:59,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:54:59,933 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:54:59,933 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:54:59,933 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:54:59,933 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:54:59,933 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:54:59,933 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:54:59,935 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:59,935 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:59,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:59,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:54:59,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:54:59,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:59,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:59,935 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:59,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:59,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:54:59,935 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:59,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:59,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:59,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:54:59,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:59,936 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:54:59,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:54:59,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,938 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,938 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:59,938 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:54:59,938 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:54:59,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:54:59,938 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:54:59,938 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:54:59,938 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:59,938 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:54:59,938 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:54:59,938 DEBUG [Thread-1666] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:54:59,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:54:59,938 DEBUG [Thread-1666] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:54:59,938 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:54:59,938 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:54:59,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:54:59,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:54:59,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:54:59,942 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:59,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 360 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1680281759940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:54:59,944 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:54:59,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40768 deadline: 1680281759942, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:00,010 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/A/2aaf8e20f90c4f62bba9a02600310c8d as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/2aaf8e20f90c4f62bba9a02600310c8d 2023-03-31 16:55:00,010 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:00,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 304 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40716 deadline: 1680281760009, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:00,012 DEBUG [Thread-1662] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4130 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8., hostname=jenkins-hbase4.apache.org,33441,1680281619375, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-31 16:55:00,015 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:00,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40748 deadline: 1680281760015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:00,015 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3ab680f7c89c77c9be754a4581df5bc8/A of 3ab680f7c89c77c9be754a4581df5bc8 into 2aaf8e20f90c4f62bba9a02600310c8d(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:55:00,016 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3ab680f7c89c77c9be754a4581df5bc8: 2023-03-31 16:55:00,016 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8., storeName=3ab680f7c89c77c9be754a4581df5bc8/A, priority=13, startTime=1680281699556; duration=0sec 2023-03-31 16:55:00,016 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:55:00,016 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 16:55:00,021 DEBUG [Thread-1664] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4139 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8., hostname=jenkins-hbase4.apache.org,33441,1680281619375, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-31 16:55:00,021 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37240 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 16:55:00,022 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): 3ab680f7c89c77c9be754a4581df5bc8/C is initiating minor compaction (all files) 2023-03-31 16:55:00,022 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3ab680f7c89c77c9be754a4581df5bc8/C in TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:55:00,022 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/8fd40085a0e84570b817f748d032fa94, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/d78c305405d8415aaadb5cf7c4b3d302, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/a54162cd97bb4bb09dfe21b2120d9ee5] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp, totalSize=36.4 K 2023-03-31 16:55:00,022 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 8fd40085a0e84570b817f748d032fa94, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=975, earliestPutTs=1680281696951 2023-03-31 16:55:00,022 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting d78c305405d8415aaadb5cf7c4b3d302, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=996, earliestPutTs=1680281698730 2023-03-31 16:55:00,023 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting a54162cd97bb4bb09dfe21b2120d9ee5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1010, earliestPutTs=1680281699350 2023-03-31 16:55:00,027 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:00,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40794 deadline: 1680281760025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:00,028 DEBUG [Thread-1660] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4147 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8., hostname=jenkins-hbase4.apache.org,33441,1680281619375, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-31 16:55:00,029 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): 3ab680f7c89c77c9be754a4581df5bc8#C#compaction#278 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:55:00,037 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=1030 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/A/8c93aca3f27a44b78d84cc1fb41ddc3e 2023-03-31 16:55:00,039 DEBUG [Thread-1666] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:00,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:00,041 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:00,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:00,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:00,042 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:00,042 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:00,042 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:00,042 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:00,043 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:00,043 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:00,045 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:00,045 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:00,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:00,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:00,045 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:00,045 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:00,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:00,046 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:00,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:00,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:00,046 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=1030 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/B/5d74219bd1e24aa78410f695da469916 2023-03-31 16:55:00,046 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:00,047 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:00,047 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:00,047 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:00,047 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:00,047 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:00,047 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:00,047 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:00,047 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:55:00,047 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. started... 2023-03-31 16:55:00,048 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. as already flushing 2023-03-31 16:55:00,048 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:55:00,048 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:00,048 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:00,048 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:00,048 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:00,048 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:00,049 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:00,049 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:00,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:00,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:00,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:00,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:00,050 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:00,050 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:00,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:00,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:00,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:00,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:00,051 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:00,051 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1dc7322a[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:00,051 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:00,051 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:00,053 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:00,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:00,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:00,053 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:00,053 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:00,053 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:00,053 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:00,053 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:00,054 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:00,054 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:00,054 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:00,054 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:00,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:00,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:00,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:00,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:00,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:00,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:00,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:00,056 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=1030 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/C/1fc7c61262ba4c35b87a52128ec1e4c9 2023-03-31 16:55:00,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:00,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:00,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:00,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:00,057 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:00,057 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:00,057 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:00,057 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:00,057 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:00,057 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:00,059 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/A/8c93aca3f27a44b78d84cc1fb41ddc3e as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/8c93aca3f27a44b78d84cc1fb41ddc3e 2023-03-31 16:55:00,060 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:00,060 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:00,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:00,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:00,060 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:00,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:00,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:00,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:00,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:00,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:00,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:00,061 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:00,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:00,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:00,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:00,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:00,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:00,063 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:00,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:00,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:00,063 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/8c93aca3f27a44b78d84cc1fb41ddc3e, entries=150, sequenceid=1030, filesize=11.9 K 2023-03-31 16:55:00,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:00,064 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/B/5d74219bd1e24aa78410f695da469916 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/5d74219bd1e24aa78410f695da469916 2023-03-31 16:55:00,065 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:00,066 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:00,066 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:00,066 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:00,066 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:00,066 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:00,066 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:00,066 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:00,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:00,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:00,066 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:00,066 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:00,066 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:00,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:00,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:00,066 DEBUG [Thread-1666] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:00,066 DEBUG [Thread-1666] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:00,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:00,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:00,068 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/5d74219bd1e24aa78410f695da469916, entries=150, sequenceid=1030, filesize=11.9 K 2023-03-31 16:55:00,069 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/B/b71d259ddca44b8dbd2c7de5c92ee511 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/b71d259ddca44b8dbd2c7de5c92ee511 2023-03-31 16:55:00,069 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/C/1fc7c61262ba4c35b87a52128ec1e4c9 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/1fc7c61262ba4c35b87a52128ec1e4c9 2023-03-31 16:55:00,073 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3ab680f7c89c77c9be754a4581df5bc8/B of 3ab680f7c89c77c9be754a4581df5bc8 into b71d259ddca44b8dbd2c7de5c92ee511(size=12.7 K), total size for store is 24.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:55:00,074 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3ab680f7c89c77c9be754a4581df5bc8: 2023-03-31 16:55:00,074 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8., storeName=3ab680f7c89c77c9be754a4581df5bc8/B, priority=13, startTime=1680281699556; duration=0sec 2023-03-31 16:55:00,074 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:55:00,075 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/1fc7c61262ba4c35b87a52128ec1e4c9, entries=150, sequenceid=1030, filesize=11.9 K 2023-03-31 16:55:00,076 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 3ab680f7c89c77c9be754a4581df5bc8 in 463ms, sequenceid=1030, compaction requested=false 2023-03-31 16:55:00,076 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3ab680f7c89c77c9be754a4581df5bc8: 2023-03-31 16:55:00,166 DEBUG [Thread-1666] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:00,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:00,168 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:00,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:00,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:00,170 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:00,170 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:00,171 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:00,171 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:00,171 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:00,171 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:00,172 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:00,172 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:00,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:00,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:00,172 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:00,172 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:00,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:00,173 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:00,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:00,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:00,173 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:00,173 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:00,173 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:00,174 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:00,174 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:00,174 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:00,174 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:00,174 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:00,174 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:55:00,174 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. started... 2023-03-31 16:55:00,174 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3ab680f7c89c77c9be754a4581df5bc8 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-31 16:55:00,174 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=A 2023-03-31 16:55:00,174 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:00,174 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=B 2023-03-31 16:55:00,174 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:00,174 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=C 2023-03-31 16:55:00,174 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:00,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on 3ab680f7c89c77c9be754a4581df5bc8 2023-03-31 16:55:00,246 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. as already flushing 2023-03-31 16:55:00,276 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:00,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 371 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1680281760274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:00,285 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:00,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40768 deadline: 1680281760284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:00,381 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:00,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 373 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1680281760378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:00,388 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:00,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40768 deadline: 1680281760386, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:00,439 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/C/4b5d5801a7f04312870eb1fd5425d988 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/4b5d5801a7f04312870eb1fd5425d988 2023-03-31 16:55:00,443 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3ab680f7c89c77c9be754a4581df5bc8/C of 3ab680f7c89c77c9be754a4581df5bc8 into 4b5d5801a7f04312870eb1fd5425d988(size=12.7 K), total size for store is 24.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:55:00,443 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3ab680f7c89c77c9be754a4581df5bc8: 2023-03-31 16:55:00,443 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8., storeName=3ab680f7c89c77c9be754a4581df5bc8/C, priority=13, startTime=1680281699556; duration=0sec 2023-03-31 16:55:00,443 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:55:00,582 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=1045 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/A/1551de7b75f94c51904f23a8e83075a6 2023-03-31 16:55:00,586 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:00,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 375 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1680281760583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:00,591 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=1045 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/B/522dc1b1091d4e3e81c9acb1b4d29446 2023-03-31 16:55:00,593 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:00,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40768 deadline: 1680281760590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:00,889 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:00,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 377 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1680281760889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:00,895 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:00,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40768 deadline: 1680281760894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:01,000 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=1045 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/C/96850842fa774465b0fd2f15d14d0886 2023-03-31 16:55:01,004 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/A/1551de7b75f94c51904f23a8e83075a6 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/1551de7b75f94c51904f23a8e83075a6 2023-03-31 16:55:01,008 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/1551de7b75f94c51904f23a8e83075a6, entries=150, sequenceid=1045, filesize=11.9 K 2023-03-31 16:55:01,008 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/B/522dc1b1091d4e3e81c9acb1b4d29446 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/522dc1b1091d4e3e81c9acb1b4d29446 2023-03-31 16:55:01,011 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/522dc1b1091d4e3e81c9acb1b4d29446, entries=150, sequenceid=1045, filesize=11.9 K 2023-03-31 16:55:01,012 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/C/96850842fa774465b0fd2f15d14d0886 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/96850842fa774465b0fd2f15d14d0886 2023-03-31 16:55:01,015 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/96850842fa774465b0fd2f15d14d0886, entries=150, sequenceid=1045, filesize=11.9 K 2023-03-31 16:55:01,016 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 3ab680f7c89c77c9be754a4581df5bc8 in 842ms, sequenceid=1045, compaction requested=true 2023-03-31 16:55:01,016 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3ab680f7c89c77c9be754a4581df5bc8: 2023-03-31 16:55:01,016 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:55:01,016 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:01,016 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:01,016 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:01,016 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:01,016 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:01,019 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:01,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:01,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:01,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:01,019 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:01,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:01,019 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:01,019 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:01,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:01,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:01,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:01,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:01,020 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:01,020 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1de1416[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:01,020 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:01,021 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:01,021 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:01,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:01,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:01,022 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:01,022 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:01,022 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:01,022 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:01,022 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:01,023 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:01,023 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:01,023 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:01,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:01,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:01,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:01,023 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:01,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:01,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:01,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:01,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:01,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:01,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:01,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:01,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:01,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:01,025 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:01,025 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:01,025 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:01,026 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:01,026 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:01,027 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:01,027 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:01,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:01,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:01,027 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:01,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:01,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:01,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:01,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:01,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:01,028 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:01,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:01,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:01,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:01,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:01,029 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:01,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:01,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:01,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:01,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:01,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:01,033 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:01,033 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:01,033 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:01,033 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:01,033 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:01,033 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:01,033 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:01,033 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:01,033 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:01,033 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:01,033 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:01,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:01,033 DEBUG [Thread-1666] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:01,033 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:01,033 DEBUG [Thread-1666] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:01,033 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:01,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:01,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:01,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:01,133 DEBUG [Thread-1666] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:01,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:01,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:01,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:01,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:01,137 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:01,137 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:01,137 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:01,137 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:01,137 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:01,137 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:01,138 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:01,138 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:01,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:01,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:01,138 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:01,139 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:01,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:01,139 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:01,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:01,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:01,139 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:01,139 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:01,140 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:01,140 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:01,140 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:01,140 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:01,140 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:01,140 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:01,140 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:55:01,140 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. started... 2023-03-31 16:55:01,140 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3ab680f7c89c77c9be754a4581df5bc8 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-31 16:55:01,140 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=A 2023-03-31 16:55:01,141 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:01,141 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=B 2023-03-31 16:55:01,141 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:01,141 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=C 2023-03-31 16:55:01,141 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:01,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on 3ab680f7c89c77c9be754a4581df5bc8 2023-03-31 16:55:01,392 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. as already flushing 2023-03-31 16:55:01,412 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:01,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 385 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1680281761408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:01,412 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:01,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40768 deadline: 1680281761408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:01,515 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:01,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 387 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1680281761513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:01,515 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:01,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40768 deadline: 1680281761513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:01,549 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=1066 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/A/99a192203327413cb219ad6690710663 2023-03-31 16:55:01,718 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:01,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 389 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1680281761717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:01,719 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:01,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40768 deadline: 1680281761717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:01,961 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=1066 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/B/bc97c36c323a4a7cb253bfa2a7557fd8 2023-03-31 16:55:02,004 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=1066 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/C/6ecc9e10289a44be882c6c8f6968e54c 2023-03-31 16:55:02,014 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/A/99a192203327413cb219ad6690710663 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/99a192203327413cb219ad6690710663 2023-03-31 16:55:02,018 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/99a192203327413cb219ad6690710663, entries=150, sequenceid=1066, filesize=11.9 K 2023-03-31 16:55:02,019 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/B/bc97c36c323a4a7cb253bfa2a7557fd8 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/bc97c36c323a4a7cb253bfa2a7557fd8 2023-03-31 16:55:02,022 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:02,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 391 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1680281762020, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,023 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:02,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40768 deadline: 1680281762021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,023 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/bc97c36c323a4a7cb253bfa2a7557fd8, entries=150, sequenceid=1066, filesize=11.9 K 2023-03-31 16:55:02,024 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/C/6ecc9e10289a44be882c6c8f6968e54c as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/6ecc9e10289a44be882c6c8f6968e54c 2023-03-31 16:55:02,027 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/6ecc9e10289a44be882c6c8f6968e54c, entries=150, sequenceid=1066, filesize=11.9 K 2023-03-31 16:55:02,028 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for 3ab680f7c89c77c9be754a4581df5bc8 in 888ms, sequenceid=1066, compaction requested=true 2023-03-31 16:55:02,028 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3ab680f7c89c77c9be754a4581df5bc8: 2023-03-31 16:55:02,028 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:55:02,028 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:02,028 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:02,028 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,028 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:02,028 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:02,031 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,031 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:02,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:02,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:02,032 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:02,032 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:02,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:02,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:02,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:02,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:02,033 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:02,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@444415e2[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:02,034 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:02,034 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:02,035 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:02,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:02,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:02,035 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:02,035 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:02,035 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:02,035 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,035 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:02,036 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:02,036 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:02,036 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,036 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:02,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:02,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:02,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:02,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:02,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:02,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:02,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:02,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:02,039 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:02,039 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:02,039 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:02,039 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:02,039 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:02,039 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:02,044 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:02,044 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:02,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:02,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:02,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:02,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:02,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:02,044 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:02,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:02,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:02,045 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:02,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:02,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:02,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:02,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:02,046 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:02,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:02,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,048 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:02,048 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:02,048 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:02,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:02,048 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:02,048 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:02,048 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:02,049 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:02,049 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:02,049 DEBUG [Thread-1666] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:02,048 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:02,049 DEBUG [Thread-1666] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:02,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:02,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:02,049 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,049 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:02,049 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:02,049 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:02,149 DEBUG [Thread-1666] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:02,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:02,151 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:02,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:02,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:02,154 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:02,154 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:02,154 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:02,154 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:02,154 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:02,154 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:02,155 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,155 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:02,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:02,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:02,156 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,156 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:02,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:02,156 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:02,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:02,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:02,157 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:02,159 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:02,159 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:02,159 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:02,159 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:02,159 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:02,160 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:02,160 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:02,160 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:55:02,160 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. started... 2023-03-31 16:55:02,160 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3ab680f7c89c77c9be754a4581df5bc8 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-31 16:55:02,160 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=A 2023-03-31 16:55:02,160 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:02,160 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=B 2023-03-31 16:55:02,160 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:02,161 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=C 2023-03-31 16:55:02,161 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:02,169 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=1079 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/A/e5baf1c23f6f44aa8ce99062f35825c5 2023-03-31 16:55:02,179 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=1079 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/B/f31076716f5b44e98a2f85982a6bf846 2023-03-31 16:55:02,188 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=1079 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/C/47e714c3054542858ae5b4cb9d2b55fe 2023-03-31 16:55:02,193 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/A/e5baf1c23f6f44aa8ce99062f35825c5 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/e5baf1c23f6f44aa8ce99062f35825c5 2023-03-31 16:55:02,197 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/e5baf1c23f6f44aa8ce99062f35825c5, entries=150, sequenceid=1079, filesize=11.9 K 2023-03-31 16:55:02,198 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/B/f31076716f5b44e98a2f85982a6bf846 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/f31076716f5b44e98a2f85982a6bf846 2023-03-31 16:55:02,204 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/f31076716f5b44e98a2f85982a6bf846, entries=150, sequenceid=1079, filesize=11.9 K 2023-03-31 16:55:02,205 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/C/47e714c3054542858ae5b4cb9d2b55fe as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/47e714c3054542858ae5b4cb9d2b55fe 2023-03-31 16:55:02,209 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/47e714c3054542858ae5b4cb9d2b55fe, entries=150, sequenceid=1079, filesize=11.9 K 2023-03-31 16:55:02,209 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=0 B/0 for 3ab680f7c89c77c9be754a4581df5bc8 in 49ms, sequenceid=1079, compaction requested=true 2023-03-31 16:55:02,210 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3ab680f7c89c77c9be754a4581df5bc8: 2023-03-31 16:55:02,210 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:55:02,210 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:02,210 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:02,210 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,210 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:02,210 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:02,213 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,213 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:02,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:02,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:02,213 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:02,213 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:02,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:02,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:02,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:02,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:02,214 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:02,214 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c037af6[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:02,214 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:02,214 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:02,216 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:02,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:02,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:02,216 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:02,216 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:02,216 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:02,216 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,216 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:02,217 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,217 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:02,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:02,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:02,217 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:02,217 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:02,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:02,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:02,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:02,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:02,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:02,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:02,220 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:02,220 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:02,220 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:02,220 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:02,220 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:02,220 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:02,222 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:02,222 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:02,222 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:02,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:02,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:02,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:02,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:02,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:02,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:02,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:02,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:02,223 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:02,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:02,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:02,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:02,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,224 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:02,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:02,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,228 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,228 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:02,228 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:02,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:02,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:02,228 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:02,228 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:02,228 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:02,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:02,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:02,228 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,229 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:02,229 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:02,229 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:02,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:02,229 DEBUG [Thread-1666] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:02,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:02,229 DEBUG [Thread-1666] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:02,329 DEBUG [Thread-1666] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:02,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:02,334 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:02,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:02,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:02,335 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:02,335 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:02,335 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:02,335 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:02,335 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:02,336 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:02,337 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:02,337 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:02,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:02,337 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,337 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:02,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:02,337 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:02,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:02,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:02,338 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:02,338 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:02,338 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:02,338 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:02,338 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:02,338 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:02,338 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:02,338 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:02,339 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:55:02,339 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. started... 2023-03-31 16:55:02,339 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3ab680f7c89c77c9be754a4581df5bc8: 2023-03-31 16:55:02,339 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:55:02,339 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:02,339 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:02,339 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,339 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:02,339 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:02,340 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:02,340 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:02,340 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:02,341 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:02,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:02,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:02,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:02,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:02,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:02,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:02,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7871453e[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:02,342 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:02,342 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:02,343 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:02,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:02,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:02,343 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,344 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:02,344 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:02,344 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:02,344 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:02,346 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,346 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:02,346 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:02,346 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:02,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:02,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:02,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:02,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:02,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:02,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:02,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:02,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:02,348 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:02,348 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:02,348 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:02,348 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:02,348 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:02,348 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:02,350 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:02,350 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:02,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:02,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:02,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:02,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:02,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:02,350 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:02,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:02,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:02,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:02,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:02,356 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:02,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:02,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:02,357 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:02,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:02,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,359 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,359 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:02,359 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:02,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:02,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:02,359 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:02,359 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:02,359 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:02,359 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:02,359 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:02,360 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,360 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:02,360 DEBUG [Thread-1666] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:02,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:02,360 DEBUG [Thread-1666] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:02,360 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:02,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:02,360 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:02,460 DEBUG [Thread-1666] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:02,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:02,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:02,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:02,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:02,463 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:02,463 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:02,463 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:02,463 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:02,464 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:02,464 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:02,465 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,465 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:02,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:02,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:02,465 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,465 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:02,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:02,466 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:02,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:02,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:02,466 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:02,466 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:02,466 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:02,466 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:02,466 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:02,466 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:02,466 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:02,466 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:02,467 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:55:02,467 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. started... 2023-03-31 16:55:02,467 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3ab680f7c89c77c9be754a4581df5bc8: 2023-03-31 16:55:02,467 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:55:02,467 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:02,467 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:02,467 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,467 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:02,467 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:02,470 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,470 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:02,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:02,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:02,470 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:02,470 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:02,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:02,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:02,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:02,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:02,472 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:02,472 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9fe9082[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:02,472 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:02,472 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:02,473 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:02,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:02,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:02,473 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:02,473 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:02,473 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:02,473 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,473 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:02,474 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:02,474 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:02,474 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:02,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:02,474 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:02,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:02,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:02,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:02,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:02,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:02,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:02,477 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:02,477 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:02,477 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:02,477 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:02,477 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:02,477 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:02,478 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:02,478 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:02,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:02,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:02,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:02,478 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:02,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:02,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:02,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:02,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:02,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:02,479 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:02,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:02,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:02,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:02,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,480 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:02,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:02,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,485 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,486 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:02,486 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:02,486 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:02,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:02,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:02,486 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:02,486 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,486 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:02,486 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:02,486 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:02,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:02,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:02,486 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:02,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:02,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:02,486 DEBUG [Thread-1666] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:02,486 DEBUG [Thread-1666] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:02,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on 3ab680f7c89c77c9be754a4581df5bc8 2023-03-31 16:55:02,538 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3ab680f7c89c77c9be754a4581df5bc8 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-31 16:55:02,538 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=A 2023-03-31 16:55:02,538 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:02,538 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=B 2023-03-31 16:55:02,538 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:02,538 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=C 2023-03-31 16:55:02,538 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:02,548 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=1091 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/A/7c8732d0582f46abbad4387b9e99b01d 2023-03-31 16:55:02,586 DEBUG [Thread-1666] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:02,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:02,589 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:02,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:02,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:02,599 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=1091 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/B/bda9a8d5b4134a5ba304f6ce93b0c5df 2023-03-31 16:55:02,599 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:02,599 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:02,599 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:02,600 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:02,600 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:02,600 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:02,601 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:02,601 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:02,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:02,601 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,601 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:02,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:02,602 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:02,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:02,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:02,603 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:02,603 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:02,603 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:02,603 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:02,603 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:02,604 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:02,604 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:02,604 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:55:02,604 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. started... 2023-03-31 16:55:02,604 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. as already flushing 2023-03-31 16:55:02,604 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:55:02,604 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:02,604 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:02,604 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,608 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:02,608 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:02,610 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,610 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:02,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:02,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:02,610 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:02,610 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:02,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:02,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:02,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:02,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:02,612 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:02,612 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6df16f5a[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:02,612 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:02,612 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:02,613 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:02,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:02,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:02,614 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:02,614 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:02,614 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:02,614 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,614 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:02,615 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:02,615 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:02,615 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,615 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:02,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:02,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:02,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:02,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:02,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:02,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:02,617 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680281702548, max timestamp=1680281702588Num uniques 859; 2023-03-31 16:55:02,617 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 16:55:02,617 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680281702548, max timestamp=1680281702588Num uniques 859; 2023-03-31 16:55:02,617 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 16:55:02,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:02,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:02,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:02,618 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:02,618 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:02,618 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:02,618 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680281702548, max timestamp=1680281702587Num uniques 859; 2023-03-31 16:55:02,618 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:02,618 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:02,618 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 16:55:02,620 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:02,620 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:02,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:02,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:02,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:02,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:02,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:02,620 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:02,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:02,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:02,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:02,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:02,621 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:02,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:02,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:02,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,622 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:02,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:02,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,625 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:02,625 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:02,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:02,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:02,625 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:02,625 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:02,625 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:02,625 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,625 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:02,625 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:02,625 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,625 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:02,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:02,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:02,625 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:02,625 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:02,626 DEBUG [Thread-1666] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:02,626 DEBUG [Thread-1666] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:02,627 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=1091 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/C/1555bd52f2b54ddfbbccb2e744fd7669 2023-03-31 16:55:02,631 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/A/7c8732d0582f46abbad4387b9e99b01d as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/7c8732d0582f46abbad4387b9e99b01d 2023-03-31 16:55:02,636 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/7c8732d0582f46abbad4387b9e99b01d, entries=150, sequenceid=1091, filesize=11.9 K 2023-03-31 16:55:02,637 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/B/bda9a8d5b4134a5ba304f6ce93b0c5df as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/bda9a8d5b4134a5ba304f6ce93b0c5df 2023-03-31 16:55:02,640 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/bda9a8d5b4134a5ba304f6ce93b0c5df, entries=150, sequenceid=1091, filesize=11.9 K 2023-03-31 16:55:02,641 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/C/1555bd52f2b54ddfbbccb2e744fd7669 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/1555bd52f2b54ddfbbccb2e744fd7669 2023-03-31 16:55:02,645 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/1555bd52f2b54ddfbbccb2e744fd7669, entries=150, sequenceid=1091, filesize=11.9 K 2023-03-31 16:55:02,646 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=100.63 KB/103050 for 3ab680f7c89c77c9be754a4581df5bc8 in 108ms, sequenceid=1091, compaction requested=true 2023-03-31 16:55:02,646 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3ab680f7c89c77c9be754a4581df5bc8: 2023-03-31 16:55:02,646 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:55:02,646 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-31 16:55:02,646 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-31 16:55:02,646 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:55:02,646 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:55:02,648 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73900 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-31 16:55:02,648 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1912): 3ab680f7c89c77c9be754a4581df5bc8/A is initiating minor compaction (all files) 2023-03-31 16:55:02,648 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3ab680f7c89c77c9be754a4581df5bc8/A in TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:55:02,649 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/2aaf8e20f90c4f62bba9a02600310c8d, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/8c93aca3f27a44b78d84cc1fb41ddc3e, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/1551de7b75f94c51904f23a8e83075a6, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/99a192203327413cb219ad6690710663, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/e5baf1c23f6f44aa8ce99062f35825c5, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/7c8732d0582f46abbad4387b9e99b01d] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp, totalSize=72.2 K 2023-03-31 16:55:02,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on 3ab680f7c89c77c9be754a4581df5bc8 2023-03-31 16:55:02,649 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3ab680f7c89c77c9be754a4581df5bc8 3/3 column families, dataSize=107.34 KB heapSize=305.30 KB 2023-03-31 16:55:02,649 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73900 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-31 16:55:02,649 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): 3ab680f7c89c77c9be754a4581df5bc8/B is initiating minor compaction (all files) 2023-03-31 16:55:02,649 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 2aaf8e20f90c4f62bba9a02600310c8d, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=1010, earliestPutTs=1680281699350 2023-03-31 16:55:02,649 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3ab680f7c89c77c9be754a4581df5bc8/B in TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:55:02,649 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/b71d259ddca44b8dbd2c7de5c92ee511, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/5d74219bd1e24aa78410f695da469916, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/522dc1b1091d4e3e81c9acb1b4d29446, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/bc97c36c323a4a7cb253bfa2a7557fd8, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/f31076716f5b44e98a2f85982a6bf846, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/bda9a8d5b4134a5ba304f6ce93b0c5df] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp, totalSize=72.2 K 2023-03-31 16:55:02,649 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=A 2023-03-31 16:55:02,649 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:55:02,649 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 8c93aca3f27a44b78d84cc1fb41ddc3e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1030, earliestPutTs=1680281699498 2023-03-31 16:55:02,649 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=B 2023-03-31 16:55:02,649 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting b71d259ddca44b8dbd2c7de5c92ee511, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=1010, earliestPutTs=1680281699350 2023-03-31 16:55:02,649 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:55:02,650 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=C 2023-03-31 16:55:02,650 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:55:02,650 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 1551de7b75f94c51904f23a8e83075a6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1045, earliestPutTs=1680281699631 2023-03-31 16:55:02,650 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 5d74219bd1e24aa78410f695da469916, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1030, earliestPutTs=1680281699498 2023-03-31 16:55:02,650 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 99a192203327413cb219ad6690710663, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1066, earliestPutTs=1680281700271 2023-03-31 16:55:02,650 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 522dc1b1091d4e3e81c9acb1b4d29446, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1045, earliestPutTs=1680281699631 2023-03-31 16:55:02,650 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting e5baf1c23f6f44aa8ce99062f35825c5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1079, earliestPutTs=1680281701403 2023-03-31 16:55:02,651 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting bc97c36c323a4a7cb253bfa2a7557fd8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1066, earliestPutTs=1680281700271 2023-03-31 16:55:02,651 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 7c8732d0582f46abbad4387b9e99b01d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1091, earliestPutTs=1680281702532 2023-03-31 16:55:02,651 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting f31076716f5b44e98a2f85982a6bf846, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1079, earliestPutTs=1680281701403 2023-03-31 16:55:02,652 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting bda9a8d5b4134a5ba304f6ce93b0c5df, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1091, earliestPutTs=1680281702532 2023-03-31 16:55:02,666 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3ab680f7c89c77c9be754a4581df5bc8#A#compaction#294 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:55:02,666 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): 3ab680f7c89c77c9be754a4581df5bc8#B#compaction#295 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:55:02,684 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:02,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 297 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40768 deadline: 1680281762677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,684 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:02,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 417 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1680281762677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,726 DEBUG [Thread-1666] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:02,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:02,728 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:02,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:02,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:02,732 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:02,732 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:02,733 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:02,733 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:02,733 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:02,733 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:02,735 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:02,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:02,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:02,735 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:02,736 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,736 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:02,736 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:02,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:02,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:02,737 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:02,737 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:02,737 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:02,737 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:02,737 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:02,737 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:02,737 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:02,737 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:02,737 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:55:02,737 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. started... 2023-03-31 16:55:02,738 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. as already flushing 2023-03-31 16:55:02,738 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:55:02,738 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:02,738 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:02,738 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,738 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:02,738 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:02,739 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:02,739 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:02,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:02,740 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:02,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:02,740 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:02,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:02,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:02,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:02,741 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:02,741 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a627cd8[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:02,741 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:02,741 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:02,743 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:02,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:02,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:02,743 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:02,743 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:02,743 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:02,743 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,743 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:02,744 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:02,744 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:02,744 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:02,744 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:02,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:02,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:02,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:02,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:02,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:02,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:02,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:02,748 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:02,748 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:02,748 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:02,748 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:02,748 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:02,748 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:02,751 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:02,751 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:02,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:02,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:02,751 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:02,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:02,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:02,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:02,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:02,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:02,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:02,752 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:02,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:02,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:02,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:02,754 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:02,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:02,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,755 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,756 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:02,756 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:02,756 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:02,756 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:02,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:02,756 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:02,756 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:02,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:02,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:02,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:02,756 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,756 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:02,756 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:02,756 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:02,756 DEBUG [Thread-1666] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:02,756 DEBUG [Thread-1666] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:02,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:02,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:02,787 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:02,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 419 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1680281762785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,787 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:02,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 299 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40768 deadline: 1680281762785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,856 DEBUG [Thread-1666] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:02,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:02,858 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:02,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:02,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:02,860 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:02,860 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:02,860 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:02,860 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:02,860 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:02,860 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:02,861 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,861 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:02,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:02,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:02,861 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,861 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:02,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:02,862 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:02,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:02,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:02,862 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:02,863 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:02,863 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:02,863 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:02,863 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:02,863 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:02,863 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:02,863 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:02,863 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:55:02,863 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. started... 2023-03-31 16:55:02,863 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. as already flushing 2023-03-31 16:55:02,863 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:55:02,863 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:02,863 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:02,863 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,863 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:02,863 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:02,865 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:02,865 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:02,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:02,865 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:02,865 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:02,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:02,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:02,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:02,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:02,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:02,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58c8941f[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:02,866 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:02,867 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:02,868 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:02,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:02,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:02,868 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,868 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:02,868 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:02,869 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:02,869 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:02,870 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:02,870 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:02,870 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,870 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:02,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:02,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:02,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:02,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:02,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:02,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:02,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:02,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:02,872 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:02,872 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:02,872 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:02,872 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:02,872 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:02,873 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:02,874 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:02,874 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:02,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:02,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:02,874 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:02,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:02,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:02,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:02,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:02,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:02,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:02,875 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:02,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:02,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:02,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:02,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,876 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:02,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:02,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,877 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,877 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:02,877 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:02,878 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:02,878 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,878 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:02,878 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:02,878 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:02,878 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:02,878 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:02,878 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:02,878 DEBUG [Thread-1666] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:02,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:02,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:02,878 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:02,878 DEBUG [Thread-1666] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:02,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:02,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:02,978 DEBUG [Thread-1666] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:02,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:02,981 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:02,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:02,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:02,982 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:02,982 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:02,982 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:02,982 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:02,982 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:02,982 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:02,987 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,987 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:02,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:02,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:02,988 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,988 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:02,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:02,988 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:02,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:02,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:02,989 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:02,989 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:02,989 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:02,989 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:02,989 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:02,989 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:02,989 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:02,989 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:02,989 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:55:02,989 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. started... 2023-03-31 16:55:02,989 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. as already flushing 2023-03-31 16:55:02,989 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:55:02,989 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:02,989 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:02,989 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,989 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:02,989 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:02,990 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:02,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 301 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40768 deadline: 1680281762988, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,991 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:02,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 421 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1680281762989, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,991 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:02,991 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:02,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:02,991 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:02,991 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:02,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:02,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:02,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:02,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:02,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:02,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25cf7504[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:02,993 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:02,993 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:02,994 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:02,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:02,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:02,995 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:02,995 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:02,995 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:02,995 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,995 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:02,997 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,997 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:02,997 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:02,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:02,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:02,997 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:02,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:02,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:02,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:02,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:02,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:02,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:02,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:03,000 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:03,000 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:03,000 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:03,000 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:03,000 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:03,000 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:03,002 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:03,002 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:03,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:03,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:03,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:03,002 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:03,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:03,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:03,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:03,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:03,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:03,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:03,004 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:03,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:03,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:03,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,007 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:03,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:03,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,010 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:03,010 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:03,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:03,010 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:03,010 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:03,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:03,010 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:03,011 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,011 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:03,011 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:03,011 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,011 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:03,011 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:03,011 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:03,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:03,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:03,011 DEBUG [Thread-1666] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:03,011 DEBUG [Thread-1666] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:03,062 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=1124 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/A/b2c69c42d64a48d28f15d77bb102a09d 2023-03-31 16:55:03,087 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/B/f1aacfb5e3b44e6284af625b43b87646 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/f1aacfb5e3b44e6284af625b43b87646 2023-03-31 16:55:03,091 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/A/b0d180f5c10242389f19ac6575f710e6 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/b0d180f5c10242389f19ac6575f710e6 2023-03-31 16:55:03,094 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 3ab680f7c89c77c9be754a4581df5bc8/B of 3ab680f7c89c77c9be754a4581df5bc8 into f1aacfb5e3b44e6284af625b43b87646(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:55:03,094 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3ab680f7c89c77c9be754a4581df5bc8: 2023-03-31 16:55:03,094 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8., storeName=3ab680f7c89c77c9be754a4581df5bc8/B, priority=10, startTime=1680281702646; duration=0sec 2023-03-31 16:55:03,094 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:55:03,094 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-31 16:55:03,098 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73900 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-31 16:55:03,098 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): 3ab680f7c89c77c9be754a4581df5bc8/C is initiating minor compaction (all files) 2023-03-31 16:55:03,098 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3ab680f7c89c77c9be754a4581df5bc8/C in TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:55:03,098 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/4b5d5801a7f04312870eb1fd5425d988, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/1fc7c61262ba4c35b87a52128ec1e4c9, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/96850842fa774465b0fd2f15d14d0886, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/6ecc9e10289a44be882c6c8f6968e54c, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/47e714c3054542858ae5b4cb9d2b55fe, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/1555bd52f2b54ddfbbccb2e744fd7669] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp, totalSize=72.2 K 2023-03-31 16:55:03,099 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 4b5d5801a7f04312870eb1fd5425d988, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=1010, earliestPutTs=1680281699350 2023-03-31 16:55:03,099 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 3ab680f7c89c77c9be754a4581df5bc8/A of 3ab680f7c89c77c9be754a4581df5bc8 into b0d180f5c10242389f19ac6575f710e6(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:55:03,099 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3ab680f7c89c77c9be754a4581df5bc8: 2023-03-31 16:55:03,099 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8., storeName=3ab680f7c89c77c9be754a4581df5bc8/A, priority=10, startTime=1680281702646; duration=0sec 2023-03-31 16:55:03,099 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 1fc7c61262ba4c35b87a52128ec1e4c9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1030, earliestPutTs=1680281699498 2023-03-31 16:55:03,099 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:55:03,100 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 96850842fa774465b0fd2f15d14d0886, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1045, earliestPutTs=1680281699631 2023-03-31 16:55:03,100 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 6ecc9e10289a44be882c6c8f6968e54c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1066, earliestPutTs=1680281700271 2023-03-31 16:55:03,101 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 47e714c3054542858ae5b4cb9d2b55fe, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1079, earliestPutTs=1680281701403 2023-03-31 16:55:03,101 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 1555bd52f2b54ddfbbccb2e744fd7669, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1091, earliestPutTs=1680281702532 2023-03-31 16:55:03,112 DEBUG [Thread-1666] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:03,112 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): 3ab680f7c89c77c9be754a4581df5bc8#C#compaction#297 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:55:03,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:03,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:03,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:03,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:03,115 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:03,115 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:03,115 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:03,115 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:03,116 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:03,116 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:03,118 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,118 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:03,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:03,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:03,118 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,118 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:03,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:03,119 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:03,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:03,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:03,119 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:03,120 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:03,120 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:03,120 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:03,120 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:03,120 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:03,124 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:03,124 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:03,124 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:55:03,124 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. started... 2023-03-31 16:55:03,124 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. as already flushing 2023-03-31 16:55:03,124 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:55:03,124 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:03,124 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:03,124 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,124 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:03,124 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:03,127 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:03,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:03,127 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:03,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:03,127 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:03,127 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:03,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:03,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:03,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:03,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:03,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43a25558[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:03,128 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:03,128 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:03,130 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:03,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:03,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:03,130 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:03,130 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:03,130 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:03,130 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,130 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:03,131 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,131 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:03,131 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:03,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:03,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:03,131 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:03,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:03,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:03,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:03,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:03,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:03,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:03,133 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:03,133 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:03,133 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:03,134 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:03,134 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:03,134 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:03,135 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:03,135 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:03,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:03,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:03,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:03,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:03,135 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:03,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:03,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:03,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:03,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:03,136 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:03,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:03,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:03,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:03,137 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:03,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:03,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,140 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,140 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:03,140 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:03,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:03,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:03,140 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:03,140 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:03,140 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:03,140 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,140 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:03,140 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:03,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:03,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:03,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:03,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:03,140 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:03,141 DEBUG [Thread-1666] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:03,141 DEBUG [Thread-1666] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:03,241 DEBUG [Thread-1666] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:03,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:03,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:03,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:03,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:03,244 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:03,244 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:03,244 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:03,244 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:03,245 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:03,245 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:03,246 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,246 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:03,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:03,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:03,246 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,246 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:03,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:03,247 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:03,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:03,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:03,247 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:03,247 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:03,247 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:03,247 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:03,247 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:03,247 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:03,248 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:03,248 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:03,248 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:55:03,248 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. started... 2023-03-31 16:55:03,248 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. as already flushing 2023-03-31 16:55:03,248 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:55:03,248 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:03,248 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:03,248 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,248 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:03,248 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:03,250 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:03,250 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:03,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:03,250 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:03,250 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:03,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:03,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:03,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:03,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:03,251 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:03,251 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:03,251 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a1f6548[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:03,251 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:03,253 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:03,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:03,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:03,253 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:03,253 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:03,253 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:03,253 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,253 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:03,255 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,255 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:03,255 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:03,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:03,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:03,255 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:03,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:03,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:03,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:03,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:03,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:03,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:03,258 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:03,258 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:03,258 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:03,258 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:03,258 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:03,258 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:03,259 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:03,259 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:03,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:03,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:03,259 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:03,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:03,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:03,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:03,260 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:03,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:03,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:03,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:03,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:03,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:03,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:03,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:03,261 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:03,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,263 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,263 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:03,263 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:03,263 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:03,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:03,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:03,263 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:03,263 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:03,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:03,263 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:03,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:03,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:03,264 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:03,264 DEBUG [Thread-1666] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:03,264 DEBUG [Thread-1666] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:03,264 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:03,264 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:03,294 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:03,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 423 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1680281763292, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,294 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:03,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 303 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40768 deadline: 1680281763292, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,364 DEBUG [Thread-1666] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:03,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:03,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:03,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:03,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:03,367 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:03,367 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:03,367 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:03,367 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:03,368 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:03,368 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:03,369 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,369 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:03,369 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,369 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:03,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:03,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:03,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:03,372 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:03,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:03,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:03,373 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:03,379 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:03,379 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:03,379 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:03,379 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:03,379 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:03,379 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:03,379 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:03,380 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:55:03,380 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. started... 2023-03-31 16:55:03,380 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. as already flushing 2023-03-31 16:55:03,380 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:55:03,380 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:03,380 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:03,380 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,380 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:03,380 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:03,385 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:03,385 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:03,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:03,385 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:03,385 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:03,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:03,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:03,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:03,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:03,386 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:03,386 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66eb6e78[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:03,386 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:03,386 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:03,388 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:03,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:03,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:03,388 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:03,388 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:03,388 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,388 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:03,388 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:03,390 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:03,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:03,390 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:03,390 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:03,390 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:03,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:03,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:03,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:03,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:03,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:03,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:03,392 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:03,392 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:03,392 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:03,392 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:03,392 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:03,392 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:03,394 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:03,394 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:03,394 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:03,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:03,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:03,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:03,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:03,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:03,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:03,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:03,394 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:03,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:03,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:03,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:03,396 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:03,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:03,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:03,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,400 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:03,400 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,400 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:03,400 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:03,400 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,400 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:03,400 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:03,400 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:03,400 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:03,400 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:03,400 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:03,400 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:03,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:03,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:03,400 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:03,400 DEBUG [Thread-1666] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:03,401 DEBUG [Thread-1666] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:03,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:03,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:03,473 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=1124 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/B/250ebfce1e274dbb9be3ef5ad7fea2de 2023-03-31 16:55:03,484 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=1124 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/C/a4d616a3cf1c495bb2334a14d1699c58 2023-03-31 16:55:03,489 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/A/b2c69c42d64a48d28f15d77bb102a09d as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/b2c69c42d64a48d28f15d77bb102a09d 2023-03-31 16:55:03,493 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/b2c69c42d64a48d28f15d77bb102a09d, entries=200, sequenceid=1124, filesize=14.3 K 2023-03-31 16:55:03,494 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/B/250ebfce1e274dbb9be3ef5ad7fea2de as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/250ebfce1e274dbb9be3ef5ad7fea2de 2023-03-31 16:55:03,497 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/250ebfce1e274dbb9be3ef5ad7fea2de, entries=150, sequenceid=1124, filesize=11.9 K 2023-03-31 16:55:03,498 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/C/a4d616a3cf1c495bb2334a14d1699c58 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/a4d616a3cf1c495bb2334a14d1699c58 2023-03-31 16:55:03,501 DEBUG [Thread-1666] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:03,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:03,502 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/a4d616a3cf1c495bb2334a14d1699c58, entries=150, sequenceid=1124, filesize=11.9 K 2023-03-31 16:55:03,503 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~305.25 KB/312576, currentSize=73.80 KB/75570 for 3ab680f7c89c77c9be754a4581df5bc8 in 854ms, sequenceid=1124, compaction requested=false 2023-03-31 16:55:03,503 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3ab680f7c89c77c9be754a4581df5bc8: 2023-03-31 16:55:03,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:03,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:03,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:03,505 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:03,505 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:03,505 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:03,505 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:03,505 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:03,505 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:03,507 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:03,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:03,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:03,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:03,507 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:03,508 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:03,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:03,508 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:03,508 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,508 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:03,510 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:03,510 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:03,510 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:03,510 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:03,510 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:03,511 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:03,511 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:03,511 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:55:03,511 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. started... 2023-03-31 16:55:03,511 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3ab680f7c89c77c9be754a4581df5bc8 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-31 16:55:03,511 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=A 2023-03-31 16:55:03,511 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:03,511 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=B 2023-03-31 16:55:03,511 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:03,511 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=C 2023-03-31 16:55:03,512 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:03,529 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/C/58a795a8bd28468a9f422e74ca6ba07f as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/58a795a8bd28468a9f422e74ca6ba07f 2023-03-31 16:55:03,542 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 3ab680f7c89c77c9be754a4581df5bc8/C of 3ab680f7c89c77c9be754a4581df5bc8 into 58a795a8bd28468a9f422e74ca6ba07f(size=12.9 K), total size for store is 24.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:55:03,542 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3ab680f7c89c77c9be754a4581df5bc8: 2023-03-31 16:55:03,542 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8., storeName=3ab680f7c89c77c9be754a4581df5bc8/C, priority=10, startTime=1680281702646; duration=0sec 2023-03-31 16:55:03,543 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:55:03,561 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=1140 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/A/58dd9207b4844db7b35bb2979cd44063 2023-03-31 16:55:03,608 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=1140 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/B/039c3c4caffb4a89b79d6d9a8e399fab 2023-03-31 16:55:03,656 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=1140 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/C/54706c23a3c04e54b5c67d1dd8ba1c2c 2023-03-31 16:55:03,662 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/A/58dd9207b4844db7b35bb2979cd44063 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/58dd9207b4844db7b35bb2979cd44063 2023-03-31 16:55:03,667 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/58dd9207b4844db7b35bb2979cd44063, entries=100, sequenceid=1140, filesize=9.5 K 2023-03-31 16:55:03,669 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/B/039c3c4caffb4a89b79d6d9a8e399fab as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/039c3c4caffb4a89b79d6d9a8e399fab 2023-03-31 16:55:03,673 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/039c3c4caffb4a89b79d6d9a8e399fab, entries=100, sequenceid=1140, filesize=9.5 K 2023-03-31 16:55:03,675 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/C/54706c23a3c04e54b5c67d1dd8ba1c2c as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/54706c23a3c04e54b5c67d1dd8ba1c2c 2023-03-31 16:55:03,687 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/54706c23a3c04e54b5c67d1dd8ba1c2c, entries=100, sequenceid=1140, filesize=9.5 K 2023-03-31 16:55:03,688 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=0 B/0 for 3ab680f7c89c77c9be754a4581df5bc8 in 177ms, sequenceid=1140, compaction requested=true 2023-03-31 16:55:03,688 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3ab680f7c89c77c9be754a4581df5bc8: 2023-03-31 16:55:03,688 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:55:03,688 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:03,688 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:03,688 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,688 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:03,688 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:03,690 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:03,690 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,690 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:03,690 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:03,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:03,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:03,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:03,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:03,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:03,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:03,692 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:03,693 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@385240f4[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:03,693 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:03,693 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:03,694 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:03,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:03,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:03,695 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:03,695 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:03,695 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:03,695 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,695 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:03,697 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,697 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:03,697 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:03,697 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:03,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:03,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:03,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:03,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:03,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:03,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:03,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:03,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:03,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:03,700 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:03,700 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:03,700 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:03,700 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:03,700 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:03,702 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:03,702 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:03,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:03,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:03,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:03,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:03,702 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:03,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:03,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:03,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:03,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:03,703 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:03,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:03,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:03,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:03,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,705 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:03,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:03,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,708 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:03,708 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:03,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:03,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:03,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:03,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:03,708 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:03,708 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,709 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:03,709 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:03,709 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,709 DEBUG [Thread-1666] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:03,709 DEBUG [Thread-1666] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:03,709 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:03,709 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:03,709 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:03,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:03,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:03,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on 3ab680f7c89c77c9be754a4581df5bc8 2023-03-31 16:55:03,803 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3ab680f7c89c77c9be754a4581df5bc8 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-31 16:55:03,803 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=A 2023-03-31 16:55:03,803 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:03,804 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=B 2023-03-31 16:55:03,804 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:03,804 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=C 2023-03-31 16:55:03,804 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:03,809 DEBUG [Thread-1666] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:03,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:03,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:03,812 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=1151 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/A/87dab3be29014a899f4cec085226ed44 2023-03-31 16:55:03,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:03,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:03,813 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:03,813 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:03,813 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:03,813 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:03,814 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:03,814 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:03,815 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:03,815 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:03,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:03,816 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,816 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:03,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:03,816 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:03,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:03,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:03,817 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:03,817 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:03,817 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:03,817 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:03,818 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:03,818 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:03,818 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:03,818 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:03,818 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:55:03,818 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. started... 2023-03-31 16:55:03,818 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. as already flushing 2023-03-31 16:55:03,818 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:55:03,818 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:03,818 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:03,818 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,819 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:03,819 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:03,821 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:03,821 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:03,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:03,822 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:03,823 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:03,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:03,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:03,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:03,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:03,835 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:03,835 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@690bda33[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:03,835 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:03,835 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:03,837 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:03,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:03,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:03,837 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:03,837 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:03,837 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:03,837 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,837 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:03,839 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,839 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:03,839 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:03,839 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:03,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:03,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:03,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:03,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:03,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:03,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:03,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:03,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:03,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:03,844 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:03,844 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:03,844 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:03,844 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:03,844 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:03,846 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:03,846 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:03,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:03,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:03,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:03,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:03,846 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:03,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:03,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:03,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:03,848 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680281703827, max timestamp=1680281703845Num uniques 859; 2023-03-31 16:55:03,848 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 16:55:03,848 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680281703827, max timestamp=1680281703845Num uniques 859; 2023-03-31 16:55:03,848 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680281703827, max timestamp=1680281703845Num uniques 859; 2023-03-31 16:55:03,848 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 16:55:03,848 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 16:55:03,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:03,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:03,849 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:03,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:03,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:03,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,851 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:03,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:03,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,855 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,855 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:03,855 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:03,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:03,855 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:03,855 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:03,855 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:03,855 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,855 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:03,855 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:03,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:03,855 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:03,855 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:03,855 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:03,856 DEBUG [Thread-1666] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:03,856 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:03,856 DEBUG [Thread-1666] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:03,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:03,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:03,882 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:03,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 444 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1680281763880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,882 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:03,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 327 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40768 deadline: 1680281763880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,956 DEBUG [Thread-1666] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:03,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:03,958 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:03,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:03,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:03,959 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:03,959 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:03,959 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:03,959 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:03,959 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:03,960 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:03,961 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,961 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:03,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:03,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:03,961 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,961 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:03,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:03,961 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:03,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:03,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:03,962 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:03,962 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:03,962 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:03,962 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:03,962 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:03,962 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:03,962 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:03,962 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:03,962 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:55:03,962 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. started... 2023-03-31 16:55:03,962 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. as already flushing 2023-03-31 16:55:03,963 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:55:03,963 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:03,963 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:03,963 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,963 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:03,963 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:03,964 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,964 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:03,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:03,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:03,964 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:03,965 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:03,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:03,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:03,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:03,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:03,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:03,966 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:03,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68bfc77f[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:03,966 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:03,967 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:03,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:03,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:03,967 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:03,967 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:03,967 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,968 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:03,968 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:03,973 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:03,973 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:03,973 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,973 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:03,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:03,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:03,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:03,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:03,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:03,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:03,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:03,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:03,975 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:03,975 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:03,975 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:03,975 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:03,975 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:03,975 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:03,977 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:03,977 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:03,977 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:03,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:03,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:03,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:03,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:03,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:03,977 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:03,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:03,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:03,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:03,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:03,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:03,978 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:03,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:03,980 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,980 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:03,980 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:03,981 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:03,980 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:03,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:03,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:03,981 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:03,981 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:03,981 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:03,981 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,981 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:03,981 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:03,981 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:03,981 DEBUG [Thread-1666] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:03,981 DEBUG [Thread-1666] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:03,981 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:03,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:03,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:03,986 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:03,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 446 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1680281763983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:03,986 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:03,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 329 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40768 deadline: 1680281763983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,028 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:04,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40748 deadline: 1680281764027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,030 DEBUG [Thread-1664] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8152 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8., hostname=jenkins-hbase4.apache.org,33441,1680281619375, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-31 16:55:04,039 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:04,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 306 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40716 deadline: 1680281764036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,040 DEBUG [Thread-1662] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8158 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8., hostname=jenkins-hbase4.apache.org,33441,1680281619375, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-31 16:55:04,058 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:04,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40794 deadline: 1680281764056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,060 DEBUG [Thread-1660] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8179 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8., hostname=jenkins-hbase4.apache.org,33441,1680281619375, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-31 16:55:04,081 DEBUG [Thread-1666] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:04,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:04,084 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:04,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:04,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:04,106 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:04,106 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:04,107 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:04,107 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:04,107 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:04,107 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:04,109 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,109 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:04,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:04,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:04,109 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,110 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:04,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:04,110 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:04,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:04,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:04,110 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:04,111 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:04,111 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:04,111 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:04,111 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:04,111 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:04,116 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:04,116 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:04,116 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:55:04,116 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. started... 2023-03-31 16:55:04,116 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. as already flushing 2023-03-31 16:55:04,116 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:55:04,116 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:04,116 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:04,116 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,116 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:04,117 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:04,119 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,119 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:04,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:04,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:04,119 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:04,119 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:04,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:04,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:04,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:04,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:04,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:04,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@559aecbe[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:04,121 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:04,121 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:04,123 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:04,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:04,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:04,123 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:04,123 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:04,123 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,123 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:04,123 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:04,124 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:04,124 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:04,124 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,124 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:04,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:04,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:04,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:04,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:04,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:04,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:04,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:04,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:04,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:04,126 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:04,127 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:04,127 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:04,127 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:04,127 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:04,128 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:04,128 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:04,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:04,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:04,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:04,128 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:04,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:04,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:04,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:04,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:04,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:04,129 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:04,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:04,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:04,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:04,131 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:04,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:04,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,134 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,134 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:04,134 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:04,134 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:04,134 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:04,134 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,134 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:04,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:04,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:04,134 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:04,134 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:04,134 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:04,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:04,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:04,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:04,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:04,135 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:04,135 DEBUG [Thread-1666] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:04,135 DEBUG [Thread-1666] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:04,189 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:04,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 448 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1680281764188, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,190 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:04,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 331 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40768 deadline: 1680281764188, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,230 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=1151 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/B/d090a5d0cc9141c4988b879f30c96556 2023-03-31 16:55:04,235 DEBUG [Thread-1666] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:04,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:04,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:04,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:04,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:04,238 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:04,238 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:04,238 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:04,238 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:04,238 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:04,238 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:04,239 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,239 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:04,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:04,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:04,240 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:04,240 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:04,244 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:04,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:04,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:04,244 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:04,252 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:04,252 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:04,252 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:04,252 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:04,252 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:04,254 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:04,254 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:04,254 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:55:04,254 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. started... 2023-03-31 16:55:04,254 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. as already flushing 2023-03-31 16:55:04,255 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:55:04,255 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:04,255 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:04,255 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,255 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:04,255 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:04,256 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,256 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:04,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:04,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:04,256 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:04,256 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:04,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:04,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:04,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:04,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:04,258 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:04,258 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f655319[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:04,258 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:04,258 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:04,259 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:04,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:04,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:04,259 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:04,259 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,259 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:04,259 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:04,259 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:04,261 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,261 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:04,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:04,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:04,261 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:04,262 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:04,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:04,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:04,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:04,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:04,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:04,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:04,264 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:04,264 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:04,264 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:04,264 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:04,264 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:04,264 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:04,265 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:04,265 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:04,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:04,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:04,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:04,265 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:04,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:04,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:04,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:04,266 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:04,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:04,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:04,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:04,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:04,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:04,267 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:04,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:04,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,269 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,269 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:04,269 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:04,269 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:04,269 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:04,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:04,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:04,269 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:04,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:04,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:04,269 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:04,269 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,270 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:04,270 DEBUG [Thread-1666] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:04,270 DEBUG [Thread-1666] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:04,270 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:04,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:04,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:04,270 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:04,370 DEBUG [Thread-1666] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:04,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:04,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:04,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:04,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:04,373 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:04,373 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:04,373 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:04,373 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:04,374 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:04,374 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:04,375 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,375 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:04,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:04,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:04,375 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,375 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:04,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:04,376 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:04,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:04,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:04,376 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:04,376 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:04,376 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:04,376 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:04,376 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:04,376 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:04,377 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:04,377 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:04,377 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:55:04,377 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. started... 2023-03-31 16:55:04,377 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. as already flushing 2023-03-31 16:55:04,377 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:55:04,377 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:04,377 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:04,377 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,377 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:04,377 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:04,380 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,380 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:04,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:04,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:04,380 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:04,380 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:04,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:04,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:04,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:04,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:04,381 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:04,381 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4888d484[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:04,381 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:04,382 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:04,383 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:04,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:04,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:04,383 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:04,383 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:04,383 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:04,383 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,383 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:04,385 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:04,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:04,385 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:04,385 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:04,385 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:04,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:04,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:04,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:04,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:04,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:04,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:04,388 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:04,388 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:04,388 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:04,388 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:04,388 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:04,388 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:04,394 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:04,394 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:04,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:04,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:04,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:04,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:04,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:04,394 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:04,394 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:04,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:04,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:04,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:04,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:04,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:04,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:04,396 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:04,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:04,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,398 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,398 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:04,398 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:04,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:04,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:04,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:04,398 DEBUG [Thread-1666] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:04,398 DEBUG [Thread-1666] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:04,398 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:04,398 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:04,398 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:04,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:04,399 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:04,399 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,399 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:04,399 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:04,399 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:04,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:04,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:04,492 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:04,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 450 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1680281764490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,493 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:04,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 333 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40768 deadline: 1680281764491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,498 DEBUG [Thread-1666] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:04,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:04,500 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:04,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:04,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:04,502 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:04,502 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:04,502 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:04,502 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:04,502 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:04,502 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:04,504 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:04,504 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:04,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:04,504 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,504 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:04,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:04,504 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:04,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:04,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:04,505 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:04,505 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:04,505 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:04,505 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:04,505 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:04,505 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:04,505 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:04,505 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:04,505 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:55:04,505 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. started... 2023-03-31 16:55:04,506 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. as already flushing 2023-03-31 16:55:04,506 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:55:04,506 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:04,506 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:04,506 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,506 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:04,506 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:04,507 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,507 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:04,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:04,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:04,508 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:04,508 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:04,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:04,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:04,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:04,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:04,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:04,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e1a76d5[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:04,509 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:04,509 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:04,511 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:04,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:04,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:04,511 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:04,511 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:04,511 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:04,511 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,511 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:04,512 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,513 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:04,513 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:04,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:04,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:04,513 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:04,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:04,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:04,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:04,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:04,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:04,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:04,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:04,515 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:04,515 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:04,515 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:04,515 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:04,515 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:04,517 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:04,517 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:04,517 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:04,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:04,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:04,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:04,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:04,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:04,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:04,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:04,517 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:04,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:04,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:04,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:04,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:04,519 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:04,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:04,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,521 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:04,521 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:04,521 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:04,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:04,521 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:04,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:04,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:04,521 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:04,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:04,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:04,521 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:04,521 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:04,521 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,521 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:04,521 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:04,521 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:04,522 DEBUG [Thread-1666] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:04,522 DEBUG [Thread-1666] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:04,622 DEBUG [Thread-1666] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:04,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:04,624 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:04,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:04,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:04,625 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:04,625 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:04,625 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:04,625 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:04,626 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:04,626 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:04,628 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,628 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:04,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:04,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:04,628 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,628 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:04,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:04,629 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:04,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:04,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:04,629 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:04,629 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:04,629 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:04,630 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:04,630 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:04,630 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:04,630 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:04,630 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:04,630 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:55:04,630 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. started... 2023-03-31 16:55:04,630 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. as already flushing 2023-03-31 16:55:04,630 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:55:04,630 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:04,630 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:04,630 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,630 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:04,630 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:04,632 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,632 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:04,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:04,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:04,633 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:04,633 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:04,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:04,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:04,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:04,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:04,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:04,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c678eb4[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:04,634 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:04,634 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:04,635 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:04,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:04,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:04,635 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:04,635 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:04,635 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,635 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:04,635 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:04,637 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,637 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:04,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:04,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:04,637 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:04,637 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:04,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:04,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:04,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:04,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:04,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:04,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:04,639 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:04,639 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:04,639 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:04,639 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:04,639 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:04,639 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:04,642 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:04,642 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:04,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:04,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:04,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:04,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:04,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:04,642 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:04,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:04,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:04,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:04,643 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:04,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:04,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:04,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:04,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,644 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:04,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:04,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,646 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:04,646 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,646 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:04,646 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:04,646 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:04,647 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:04,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:04,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:04,646 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:04,647 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:04,647 DEBUG [Thread-1666] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:04,647 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,647 DEBUG [Thread-1666] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:04,647 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:04,647 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:04,647 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:04,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:04,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:04,652 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=1151 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/C/20113e99ee6f4e898e4438af4bb3e060 2023-03-31 16:55:04,663 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/A/87dab3be29014a899f4cec085226ed44 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/87dab3be29014a899f4cec085226ed44 2023-03-31 16:55:04,668 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/87dab3be29014a899f4cec085226ed44, entries=150, sequenceid=1151, filesize=11.9 K 2023-03-31 16:55:04,669 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/B/d090a5d0cc9141c4988b879f30c96556 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/d090a5d0cc9141c4988b879f30c96556 2023-03-31 16:55:04,674 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/d090a5d0cc9141c4988b879f30c96556, entries=150, sequenceid=1151, filesize=11.9 K 2023-03-31 16:55:04,674 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/C/20113e99ee6f4e898e4438af4bb3e060 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/20113e99ee6f4e898e4438af4bb3e060 2023-03-31 16:55:04,681 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/20113e99ee6f4e898e4438af4bb3e060, entries=150, sequenceid=1151, filesize=11.9 K 2023-03-31 16:55:04,682 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 3ab680f7c89c77c9be754a4581df5bc8 in 879ms, sequenceid=1151, compaction requested=true 2023-03-31 16:55:04,682 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3ab680f7c89c77c9be754a4581df5bc8: 2023-03-31 16:55:04,682 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:55:04,682 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:55:04,682 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 16:55:04,682 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 16:55:04,682 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:55:04,683 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47288 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 16:55:04,683 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): 3ab680f7c89c77c9be754a4581df5bc8/B is initiating minor compaction (all files) 2023-03-31 16:55:04,683 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49728 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 16:55:04,683 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3ab680f7c89c77c9be754a4581df5bc8/B in TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:55:04,683 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1912): 3ab680f7c89c77c9be754a4581df5bc8/A is initiating minor compaction (all files) 2023-03-31 16:55:04,683 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/f1aacfb5e3b44e6284af625b43b87646, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/250ebfce1e274dbb9be3ef5ad7fea2de, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/039c3c4caffb4a89b79d6d9a8e399fab, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/d090a5d0cc9141c4988b879f30c96556] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp, totalSize=46.2 K 2023-03-31 16:55:04,683 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3ab680f7c89c77c9be754a4581df5bc8/A in TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:55:04,683 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/b0d180f5c10242389f19ac6575f710e6, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/b2c69c42d64a48d28f15d77bb102a09d, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/58dd9207b4844db7b35bb2979cd44063, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/87dab3be29014a899f4cec085226ed44] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp, totalSize=48.6 K 2023-03-31 16:55:04,683 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting f1aacfb5e3b44e6284af625b43b87646, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=1091, earliestPutTs=1680281702532 2023-03-31 16:55:04,684 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting b0d180f5c10242389f19ac6575f710e6, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=1091, earliestPutTs=1680281702532 2023-03-31 16:55:04,684 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 250ebfce1e274dbb9be3ef5ad7fea2de, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1124, earliestPutTs=1680281702633 2023-03-31 16:55:04,684 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting b2c69c42d64a48d28f15d77bb102a09d, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=1124, earliestPutTs=1680281702626 2023-03-31 16:55:04,684 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 039c3c4caffb4a89b79d6d9a8e399fab, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=1140, earliestPutTs=1680281702666 2023-03-31 16:55:04,684 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 58dd9207b4844db7b35bb2979cd44063, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=1140, earliestPutTs=1680281702666 2023-03-31 16:55:04,684 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting d090a5d0cc9141c4988b879f30c96556, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1151, earliestPutTs=1680281703797 2023-03-31 16:55:04,685 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 87dab3be29014a899f4cec085226ed44, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1151, earliestPutTs=1680281703797 2023-03-31 16:55:04,706 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): 3ab680f7c89c77c9be754a4581df5bc8#B#compaction#305 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:55:04,714 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3ab680f7c89c77c9be754a4581df5bc8#A#compaction#306 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:55:04,737 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/A/bc8dad9976d441f898d5c55088c585e1 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/bc8dad9976d441f898d5c55088c585e1 2023-03-31 16:55:04,741 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3ab680f7c89c77c9be754a4581df5bc8/A of 3ab680f7c89c77c9be754a4581df5bc8 into bc8dad9976d441f898d5c55088c585e1(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:55:04,742 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3ab680f7c89c77c9be754a4581df5bc8: 2023-03-31 16:55:04,742 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8., storeName=3ab680f7c89c77c9be754a4581df5bc8/A, priority=12, startTime=1680281704682; duration=0sec 2023-03-31 16:55:04,742 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:55:04,742 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 16:55:04,743 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47288 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 16:55:04,743 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1912): 3ab680f7c89c77c9be754a4581df5bc8/C is initiating minor compaction (all files) 2023-03-31 16:55:04,743 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3ab680f7c89c77c9be754a4581df5bc8/C in TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:55:04,743 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/58a795a8bd28468a9f422e74ca6ba07f, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/a4d616a3cf1c495bb2334a14d1699c58, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/54706c23a3c04e54b5c67d1dd8ba1c2c, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/20113e99ee6f4e898e4438af4bb3e060] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp, totalSize=46.2 K 2023-03-31 16:55:04,744 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 58a795a8bd28468a9f422e74ca6ba07f, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=1091, earliestPutTs=1680281702532 2023-03-31 16:55:04,744 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting a4d616a3cf1c495bb2334a14d1699c58, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1124, earliestPutTs=1680281702633 2023-03-31 16:55:04,744 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 54706c23a3c04e54b5c67d1dd8ba1c2c, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=1140, earliestPutTs=1680281702666 2023-03-31 16:55:04,745 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 20113e99ee6f4e898e4438af4bb3e060, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1151, earliestPutTs=1680281703797 2023-03-31 16:55:04,747 DEBUG [Thread-1666] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:04,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:04,749 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:04,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:04,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:04,751 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:04,751 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:04,751 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:04,751 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:04,751 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3ab680f7c89c77c9be754a4581df5bc8#C#compaction#307 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:55:04,751 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:04,751 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:04,753 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,753 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:04,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:04,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:04,753 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,753 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:04,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:04,754 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:04,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:04,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:04,754 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:04,763 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:04,763 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:04,763 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:04,763 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:04,763 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:04,763 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:04,763 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:04,763 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:55:04,763 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. started... 2023-03-31 16:55:04,764 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3ab680f7c89c77c9be754a4581df5bc8 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-03-31 16:55:04,764 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=A 2023-03-31 16:55:04,764 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:55:04,764 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=B 2023-03-31 16:55:04,764 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:55:04,765 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=C 2023-03-31 16:55:04,765 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:55:04,780 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/C/33f1d682ece44294a3e4910a2543f38c as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/33f1d682ece44294a3e4910a2543f38c 2023-03-31 16:55:04,785 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3ab680f7c89c77c9be754a4581df5bc8/C of 3ab680f7c89c77c9be754a4581df5bc8 into 33f1d682ece44294a3e4910a2543f38c(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:55:04,785 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3ab680f7c89c77c9be754a4581df5bc8: 2023-03-31 16:55:04,785 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8., storeName=3ab680f7c89c77c9be754a4581df5bc8/C, priority=12, startTime=1680281704682; duration=0sec 2023-03-31 16:55:04,785 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:55:04,787 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=1189 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/A/9748660522264f609f438d3b20078eb6 2023-03-31 16:55:04,804 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=1189 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/B/c2c2950e2772476fb3c62e7477c5cfb6 2023-03-31 16:55:04,813 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=1189 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/C/ea08b127ac704df3a5a5873bf3c55702 2023-03-31 16:55:04,817 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/A/9748660522264f609f438d3b20078eb6 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/9748660522264f609f438d3b20078eb6 2023-03-31 16:55:04,821 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/9748660522264f609f438d3b20078eb6, entries=150, sequenceid=1189, filesize=11.9 K 2023-03-31 16:55:04,822 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/B/c2c2950e2772476fb3c62e7477c5cfb6 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/c2c2950e2772476fb3c62e7477c5cfb6 2023-03-31 16:55:04,826 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/c2c2950e2772476fb3c62e7477c5cfb6, entries=150, sequenceid=1189, filesize=11.9 K 2023-03-31 16:55:04,827 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/C/ea08b127ac704df3a5a5873bf3c55702 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/ea08b127ac704df3a5a5873bf3c55702 2023-03-31 16:55:04,830 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/ea08b127ac704df3a5a5873bf3c55702, entries=150, sequenceid=1189, filesize=11.9 K 2023-03-31 16:55:04,830 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=0 B/0 for 3ab680f7c89c77c9be754a4581df5bc8 in 66ms, sequenceid=1189, compaction requested=false 2023-03-31 16:55:04,831 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3ab680f7c89c77c9be754a4581df5bc8: 2023-03-31 16:55:04,831 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:55:04,831 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:04,831 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:04,831 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,831 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:04,831 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:04,832 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:04,832 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:04,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:04,833 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:04,833 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:04,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:04,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:04,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:04,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:04,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:04,835 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@94fe3ba[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:04,835 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:04,835 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:04,837 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:04,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:04,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:04,837 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:04,837 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:04,837 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:04,837 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,837 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:04,839 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:04,839 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,839 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:04,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:04,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:04,839 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:04,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:04,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:04,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:04,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:04,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:04,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:04,842 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:04,842 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:04,842 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:04,842 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:04,842 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:04,842 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:04,844 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:04,844 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:04,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:04,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:04,844 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:04,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:04,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:04,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:04,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:04,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:04,844 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:04,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:04,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:04,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:04,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:04,846 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:04,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:04,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,852 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:04,852 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:04,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:04,852 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:04,852 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:04,852 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:04,852 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:04,853 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:04,853 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:04,853 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,853 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:04,853 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:04,853 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:04,853 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:04,853 DEBUG [Thread-1666] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:04,853 DEBUG [Thread-1666] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:04,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:04,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:04,953 DEBUG [Thread-1666] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:04,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:04,955 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:04,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:04,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:04,957 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:04,957 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:04,957 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:04,957 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:04,958 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:04,958 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:04,959 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,959 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,959 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:04,961 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:04,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:04,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:04,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:04,962 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:04,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:04,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:04,962 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:04,962 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:04,962 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:04,962 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:04,962 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:04,962 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:04,963 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:04,963 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:04,963 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:55:04,963 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. started... 2023-03-31 16:55:04,963 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3ab680f7c89c77c9be754a4581df5bc8: 2023-03-31 16:55:04,963 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:55:04,963 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:04,963 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:04,963 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,963 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:04,963 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:04,965 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,965 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:04,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:04,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:04,965 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:04,965 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:04,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:04,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:04,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:04,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:04,968 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:04,969 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ed2ae3[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:04,969 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:04,969 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:04,970 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:04,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:04,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:04,970 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:04,970 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:04,970 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:04,973 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:04,973 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:04,973 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:04,973 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,974 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:04,974 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:04,974 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:04,974 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:04,974 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:04,974 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:04,974 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:04,977 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:04,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:04,977 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:04,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:04,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:04,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:04,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:04,977 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:04,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:04,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:04,978 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:04,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:04,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:04,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:04,980 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:04,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:04,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:04,982 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:04,982 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:04,983 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,983 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:04,983 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:04,983 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:04,983 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:04,983 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:04,983 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:04,983 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:04,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:04,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:04,983 DEBUG [Thread-1666] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:04,983 DEBUG [Thread-1666] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:04,983 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:04,983 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:04,983 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:04,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:04,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:05,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on 3ab680f7c89c77c9be754a4581df5bc8 2023-03-31 16:55:05,012 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3ab680f7c89c77c9be754a4581df5bc8 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-31 16:55:05,012 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=A 2023-03-31 16:55:05,012 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:05,012 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=B 2023-03-31 16:55:05,012 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:05,012 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=C 2023-03-31 16:55:05,012 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:05,043 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1202 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/A/ef607b1205a24a01b772e99b884ce549 2023-03-31 16:55:05,058 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680281705042, max timestamp=1680281705056Num uniques 859; 2023-03-31 16:55:05,058 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680281705042, max timestamp=1680281705056Num uniques 859; 2023-03-31 16:55:05,058 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 16:55:05,058 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 16:55:05,060 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680281705051, max timestamp=1680281705056Num uniques 859; 2023-03-31 16:55:05,060 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 16:55:05,083 DEBUG [Thread-1666] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:05,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:05,086 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:05,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:05,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:05,087 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:05,087 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:05,087 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:05,087 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:05,088 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:05,088 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:05,089 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:05,089 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:05,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:05,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:05,090 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:05,090 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:05,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:05,090 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:05,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:05,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:05,090 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:05,090 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:05,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 472 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1680281765087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:05,092 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:05,092 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:05,092 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:05,092 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:05,092 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:05,092 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:05,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 355 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40768 deadline: 1680281765090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:05,100 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:05,100 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:05,100 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:55:05,100 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. started... 2023-03-31 16:55:05,100 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. as already flushing 2023-03-31 16:55:05,100 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:55:05,101 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:05,101 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:05,101 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:05,101 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:05,101 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:05,103 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:05,103 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:05,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:05,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:05,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:05,104 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:05,104 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:05,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:05,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:05,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:05,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:05,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:05,106 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:05,106 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a4c5815[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:05,106 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:05,106 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:05,108 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:05,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:05,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:05,108 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:05,108 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:05,108 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:05,108 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:05,108 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:05,110 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:05,110 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:05,110 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:05,110 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:05,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:05,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:05,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:05,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:05,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:05,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:05,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:05,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:05,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:05,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:05,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:05,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:05,113 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:05,113 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:05,114 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:05,114 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:05,114 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:05,116 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:05,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:05,116 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:05,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:05,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:05,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:05,116 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:05,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:05,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:05,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:05,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:05,117 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:05,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:05,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:05,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:05,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:05,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:05,119 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:05,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:05,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:05,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:05,124 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:05,124 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:05,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:05,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:05,125 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:05,125 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:05,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:05,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:05,125 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:05,125 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:05,125 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:05,125 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:05,125 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:05,125 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:05,125 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:05,125 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:05,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:05,126 DEBUG [Thread-1666] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:05,126 DEBUG [Thread-1666] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:05,140 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/B/3f1db28f389f4843b1b3e2372305da7f as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/3f1db28f389f4843b1b3e2372305da7f 2023-03-31 16:55:05,146 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3ab680f7c89c77c9be754a4581df5bc8/B of 3ab680f7c89c77c9be754a4581df5bc8 into 3f1db28f389f4843b1b3e2372305da7f(size=13.0 K), total size for store is 24.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:55:05,146 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3ab680f7c89c77c9be754a4581df5bc8: 2023-03-31 16:55:05,146 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8., storeName=3ab680f7c89c77c9be754a4581df5bc8/B, priority=12, startTime=1680281704682; duration=0sec 2023-03-31 16:55:05,146 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:55:05,192 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:05,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 474 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1680281765191, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:05,194 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:05,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 357 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40768 deadline: 1680281765193, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:05,226 DEBUG [Thread-1666] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:05,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:05,229 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:05,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:05,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:05,230 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:05,230 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:05,230 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:05,230 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:05,231 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:05,231 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:05,232 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:05,233 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:05,233 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:05,234 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:05,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:05,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:05,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:05,235 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:05,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:05,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:05,236 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:05,236 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:05,236 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:05,236 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:05,236 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:05,236 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:05,236 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:05,236 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:05,236 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:55:05,236 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. started... 2023-03-31 16:55:05,237 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. as already flushing 2023-03-31 16:55:05,237 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:55:05,237 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:05,237 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:05,237 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:05,237 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:05,237 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:05,238 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:05,238 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:05,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:05,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:05,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:05,239 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:05,239 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:05,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:05,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:05,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:05,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:05,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:05,240 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:05,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a3e65c6[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:05,241 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:05,241 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:05,242 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:05,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:05,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:05,242 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:05,242 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:05,242 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:05,242 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:05,242 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:05,244 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:05,244 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:05,244 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:05,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:05,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:05,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:05,244 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:05,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:05,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:05,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:05,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:05,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:05,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:05,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:05,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:05,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:05,247 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:05,247 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:05,247 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:05,247 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:05,247 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:05,249 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:05,249 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:05,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:05,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:05,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:05,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:05,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:05,249 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:05,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:05,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:05,250 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:05,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:05,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:05,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:05,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:05,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:05,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:05,251 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:05,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:05,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:05,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:05,257 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:05,257 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:05,257 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:05,257 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:05,257 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:05,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:05,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:05,257 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:05,257 DEBUG [Thread-1666] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:05,257 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:05,257 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:05,257 DEBUG [Thread-1666] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:05,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:05,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:05,257 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:05,257 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:05,257 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:05,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:05,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:05,357 DEBUG [Thread-1666] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:05,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:05,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:05,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:05,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:05,361 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:05,361 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:05,361 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:05,361 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:05,361 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:05,361 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:05,367 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:05,367 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:05,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:05,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:05,368 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:05,368 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:05,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:05,368 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:05,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:05,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:05,369 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:05,369 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:05,369 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:05,369 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:05,369 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:05,369 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:05,369 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:05,369 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:55:05,369 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. started... 2023-03-31 16:55:05,369 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:05,369 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. as already flushing 2023-03-31 16:55:05,369 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:55:05,369 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:05,369 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:05,369 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:05,370 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:05,370 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:05,371 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:05,371 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:05,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:05,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:05,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:05,371 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:05,372 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:05,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:05,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:05,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:05,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:05,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:05,373 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:05,373 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:05,373 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c9a0908[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:05,373 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:05,374 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:05,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:05,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:05,374 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:05,374 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:05,374 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:05,374 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:05,374 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:05,375 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:05,376 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:05,376 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:05,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:05,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:05,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:05,376 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:05,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:05,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:05,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:05,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:05,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:05,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:05,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:05,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:05,378 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:05,378 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:05,378 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:05,378 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:05,378 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:05,378 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:05,380 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:05,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:05,380 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:05,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:05,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:05,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:05,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:05,380 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:05,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:05,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:05,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:05,381 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:05,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:05,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:05,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:05,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:05,382 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:05,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:05,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:05,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:05,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:05,384 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:05,384 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:05,384 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:05,384 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:05,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:05,384 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:05,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:05,384 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:05,384 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:05,384 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:05,384 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:05,384 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:05,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:05,384 DEBUG [Thread-1666] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:05,384 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:05,385 DEBUG [Thread-1666] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:05,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:05,385 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:05,385 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:05,397 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:05,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 476 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1680281765394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:05,398 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:05,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 359 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40768 deadline: 1680281765396, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:05,465 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1202 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/B/76b6126c1fb8460aafee4c1bc1fca2ad 2023-03-31 16:55:05,475 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1202 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/C/34cc6ed295a24d8383b1eedfaddf46b5 2023-03-31 16:55:05,479 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/A/ef607b1205a24a01b772e99b884ce549 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/ef607b1205a24a01b772e99b884ce549 2023-03-31 16:55:05,482 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/ef607b1205a24a01b772e99b884ce549, entries=150, sequenceid=1202, filesize=11.9 K 2023-03-31 16:55:05,483 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/B/76b6126c1fb8460aafee4c1bc1fca2ad as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/76b6126c1fb8460aafee4c1bc1fca2ad 2023-03-31 16:55:05,485 DEBUG [Thread-1666] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:05,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:05,486 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/76b6126c1fb8460aafee4c1bc1fca2ad, entries=150, sequenceid=1202, filesize=11.9 K 2023-03-31 16:55:05,487 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/C/34cc6ed295a24d8383b1eedfaddf46b5 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/34cc6ed295a24d8383b1eedfaddf46b5 2023-03-31 16:55:05,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:05,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:05,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:05,488 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:05,488 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:05,488 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:05,488 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:05,489 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:05,489 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:05,490 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:05,490 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:05,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:05,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:05,490 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:05,490 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:05,491 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/34cc6ed295a24d8383b1eedfaddf46b5, entries=150, sequenceid=1202, filesize=11.9 K 2023-03-31 16:55:05,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:05,491 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:05,491 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=125.23 KB/128240 for 3ab680f7c89c77c9be754a4581df5bc8 in 479ms, sequenceid=1202, compaction requested=true 2023-03-31 16:55:05,491 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3ab680f7c89c77c9be754a4581df5bc8: 2023-03-31 16:55:05,492 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:55:05,492 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:55:05,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:05,492 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 16:55:05,492 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:55:05,492 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 16:55:05,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:05,492 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:05,492 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:05,492 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:05,492 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:05,492 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:05,492 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:05,492 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:05,493 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:05,493 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:55:05,493 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. started... 2023-03-31 16:55:05,493 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37682 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 16:55:05,493 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3ab680f7c89c77c9be754a4581df5bc8 3/3 column families, dataSize=125.23 KB heapSize=355.30 KB 2023-03-31 16:55:05,493 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): 3ab680f7c89c77c9be754a4581df5bc8/B is initiating minor compaction (all files) 2023-03-31 16:55:05,493 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3ab680f7c89c77c9be754a4581df5bc8/B in TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:55:05,493 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=A 2023-03-31 16:55:05,493 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/3f1db28f389f4843b1b3e2372305da7f, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/c2c2950e2772476fb3c62e7477c5cfb6, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/76b6126c1fb8460aafee4c1bc1fca2ad] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp, totalSize=36.8 K 2023-03-31 16:55:05,493 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:55:05,493 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=B 2023-03-31 16:55:05,493 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:55:05,493 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=C 2023-03-31 16:55:05,493 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:55:05,494 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37682 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 16:55:05,494 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 3f1db28f389f4843b1b3e2372305da7f, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=1151, earliestPutTs=1680281703797 2023-03-31 16:55:05,494 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1912): 3ab680f7c89c77c9be754a4581df5bc8/A is initiating minor compaction (all files) 2023-03-31 16:55:05,494 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3ab680f7c89c77c9be754a4581df5bc8/A in TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:55:05,494 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/bc8dad9976d441f898d5c55088c585e1, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/9748660522264f609f438d3b20078eb6, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/ef607b1205a24a01b772e99b884ce549] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp, totalSize=36.8 K 2023-03-31 16:55:05,494 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting c2c2950e2772476fb3c62e7477c5cfb6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1189, earliestPutTs=1680281703865 2023-03-31 16:55:05,495 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting bc8dad9976d441f898d5c55088c585e1, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=1151, earliestPutTs=1680281703797 2023-03-31 16:55:05,495 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 76b6126c1fb8460aafee4c1bc1fca2ad, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1202, earliestPutTs=1680281705005 2023-03-31 16:55:05,495 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 9748660522264f609f438d3b20078eb6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1189, earliestPutTs=1680281703865 2023-03-31 16:55:05,496 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting ef607b1205a24a01b772e99b884ce549, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1202, earliestPutTs=1680281705005 2023-03-31 16:55:05,502 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): 3ab680f7c89c77c9be754a4581df5bc8#B#compaction#315 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:55:05,509 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3ab680f7c89c77c9be754a4581df5bc8#A#compaction#316 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:55:05,523 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/B/e7adccabb403473fbec840e4e85be90e as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/e7adccabb403473fbec840e4e85be90e 2023-03-31 16:55:05,528 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3ab680f7c89c77c9be754a4581df5bc8/B of 3ab680f7c89c77c9be754a4581df5bc8 into e7adccabb403473fbec840e4e85be90e(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:55:05,528 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3ab680f7c89c77c9be754a4581df5bc8: 2023-03-31 16:55:05,528 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8., storeName=3ab680f7c89c77c9be754a4581df5bc8/B, priority=13, startTime=1680281705492; duration=0sec 2023-03-31 16:55:05,528 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:55:05,528 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 16:55:05,529 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37682 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 16:55:05,529 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): 3ab680f7c89c77c9be754a4581df5bc8/C is initiating minor compaction (all files) 2023-03-31 16:55:05,529 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3ab680f7c89c77c9be754a4581df5bc8/C in TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:55:05,529 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/33f1d682ece44294a3e4910a2543f38c, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/ea08b127ac704df3a5a5873bf3c55702, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/34cc6ed295a24d8383b1eedfaddf46b5] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp, totalSize=36.8 K 2023-03-31 16:55:05,530 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 33f1d682ece44294a3e4910a2543f38c, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=1151, earliestPutTs=1680281703797 2023-03-31 16:55:05,530 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting ea08b127ac704df3a5a5873bf3c55702, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1189, earliestPutTs=1680281703865 2023-03-31 16:55:05,530 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 34cc6ed295a24d8383b1eedfaddf46b5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1202, earliestPutTs=1680281705005 2023-03-31 16:55:05,536 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): 3ab680f7c89c77c9be754a4581df5bc8#C#compaction#317 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:55:05,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on 3ab680f7c89c77c9be754a4581df5bc8 2023-03-31 16:55:05,702 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. as already flushing 2023-03-31 16:55:05,717 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:05,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 482 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1680281765713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:05,717 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:05,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 365 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40768 deadline: 1680281765713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:05,820 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:05,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 367 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40768 deadline: 1680281765818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:05,820 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:05,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 484 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1680281765818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:05,910 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=1238 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/A/8901902bfc7c4d4986e2b1533fe344aa 2023-03-31 16:55:05,929 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/A/b97db0a56b87455eaa3afb8cc2386acf as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/b97db0a56b87455eaa3afb8cc2386acf 2023-03-31 16:55:05,933 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3ab680f7c89c77c9be754a4581df5bc8/A of 3ab680f7c89c77c9be754a4581df5bc8 into b97db0a56b87455eaa3afb8cc2386acf(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:55:05,933 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3ab680f7c89c77c9be754a4581df5bc8: 2023-03-31 16:55:05,933 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8., storeName=3ab680f7c89c77c9be754a4581df5bc8/A, priority=13, startTime=1680281705492; duration=0sec 2023-03-31 16:55:05,933 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:55:05,958 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/C/7eead4bbc30548fba31a7624f1c1eac7 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/7eead4bbc30548fba31a7624f1c1eac7 2023-03-31 16:55:05,962 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3ab680f7c89c77c9be754a4581df5bc8/C of 3ab680f7c89c77c9be754a4581df5bc8 into 7eead4bbc30548fba31a7624f1c1eac7(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:55:05,962 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3ab680f7c89c77c9be754a4581df5bc8: 2023-03-31 16:55:05,962 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8., storeName=3ab680f7c89c77c9be754a4581df5bc8/C, priority=13, startTime=1680281705492; duration=0sec 2023-03-31 16:55:05,962 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:55:06,021 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:06,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 369 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40768 deadline: 1680281766021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:06,022 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:06,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 486 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1680281766022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:06,320 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=1238 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/B/61f61c30c60c4cef87c35f2cd26d9e11 2023-03-31 16:55:06,327 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:06,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 371 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40768 deadline: 1680281766325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:06,327 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:06,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 488 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1680281766325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:06,338 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=1238 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/C/fdae192f025d4439822ab420380c3884 2023-03-31 16:55:06,344 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/A/8901902bfc7c4d4986e2b1533fe344aa as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/8901902bfc7c4d4986e2b1533fe344aa 2023-03-31 16:55:06,364 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/8901902bfc7c4d4986e2b1533fe344aa, entries=150, sequenceid=1238, filesize=11.9 K 2023-03-31 16:55:06,372 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/B/61f61c30c60c4cef87c35f2cd26d9e11 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/61f61c30c60c4cef87c35f2cd26d9e11 2023-03-31 16:55:06,376 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/61f61c30c60c4cef87c35f2cd26d9e11, entries=150, sequenceid=1238, filesize=11.9 K 2023-03-31 16:55:06,377 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/C/fdae192f025d4439822ab420380c3884 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/fdae192f025d4439822ab420380c3884 2023-03-31 16:55:06,380 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/fdae192f025d4439822ab420380c3884, entries=150, sequenceid=1238, filesize=11.9 K 2023-03-31 16:55:06,381 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~125.23 KB/128240, heapSize ~355.25 KB/363776, currentSize=53.67 KB/54960 for 3ab680f7c89c77c9be754a4581df5bc8 in 888ms, sequenceid=1238, compaction requested=false 2023-03-31 16:55:06,381 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3ab680f7c89c77c9be754a4581df5bc8: 2023-03-31 16:55:06,381 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:55:06,381 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:06,381 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:06,381 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:06,381 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:06,381 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:06,383 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:06,383 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:06,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:06,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:06,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:06,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:06,383 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:06,383 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:06,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:06,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:06,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:06,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:06,385 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:06,385 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@724e924a[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:06,385 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:06,385 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:06,387 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:06,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:06,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:06,388 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:06,388 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:06,388 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:06,388 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:06,388 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:06,389 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:06,389 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:06,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:06,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:06,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:06,389 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:06,389 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:06,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:06,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:06,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:06,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:06,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:06,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:06,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:06,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:06,391 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:06,391 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:06,391 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:06,391 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:06,391 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:06,391 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:06,393 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:06,393 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:06,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:06,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:06,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:06,393 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:06,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:06,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:06,394 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:06,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:06,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:06,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:06,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:06,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:06,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:06,395 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:06,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:06,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:06,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:06,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:06,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:06,398 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:06,398 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:06,398 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:06,398 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:06,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:06,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:06,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:06,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:06,398 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:06,398 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:06,398 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:06,398 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:06,398 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:06,398 DEBUG [Thread-1666] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:06,398 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:06,398 DEBUG [Thread-1666] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:06,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:06,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:06,399 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:06,498 DEBUG [Thread-1666] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:06,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:06,502 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:06,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:06,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:06,524 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:06,524 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:06,524 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:06,524 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:06,524 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:06,525 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:06,527 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:06,527 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:06,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:06,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:06,528 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:06,528 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:06,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:06,528 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:06,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:06,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:06,528 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:06,532 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:06,532 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:06,532 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:06,532 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:06,532 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:06,552 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:06,552 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:06,552 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:55:06,552 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. started... 2023-03-31 16:55:06,552 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3ab680f7c89c77c9be754a4581df5bc8 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-31 16:55:06,552 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=A 2023-03-31 16:55:06,553 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:06,553 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=B 2023-03-31 16:55:06,553 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:06,553 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=C 2023-03-31 16:55:06,553 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:06,583 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1252 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/A/389d272996f54996a8337f3c06d0bb51 2023-03-31 16:55:06,610 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1252 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/B/9e77f7fe217f4c3b899fbf8d96a5bdee 2023-03-31 16:55:06,681 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1252 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/C/673908c6b2c9419c8b58f5e16d13ef53 2023-03-31 16:55:06,685 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/A/389d272996f54996a8337f3c06d0bb51 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/389d272996f54996a8337f3c06d0bb51 2023-03-31 16:55:06,689 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/389d272996f54996a8337f3c06d0bb51, entries=150, sequenceid=1252, filesize=11.9 K 2023-03-31 16:55:06,690 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/B/9e77f7fe217f4c3b899fbf8d96a5bdee as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/9e77f7fe217f4c3b899fbf8d96a5bdee 2023-03-31 16:55:06,694 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/9e77f7fe217f4c3b899fbf8d96a5bdee, entries=150, sequenceid=1252, filesize=11.9 K 2023-03-31 16:55:06,695 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/C/673908c6b2c9419c8b58f5e16d13ef53 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/673908c6b2c9419c8b58f5e16d13ef53 2023-03-31 16:55:06,700 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/673908c6b2c9419c8b58f5e16d13ef53, entries=150, sequenceid=1252, filesize=11.9 K 2023-03-31 16:55:06,701 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=0 B/0 for 3ab680f7c89c77c9be754a4581df5bc8 in 149ms, sequenceid=1252, compaction requested=true 2023-03-31 16:55:06,701 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3ab680f7c89c77c9be754a4581df5bc8: 2023-03-31 16:55:06,701 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:55:06,701 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:06,701 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:06,701 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:06,704 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:06,704 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:06,706 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:06,706 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:06,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:06,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:06,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:06,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:06,706 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:06,706 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:06,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:06,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:06,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:06,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:06,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:06,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1aaaf0e6[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:06,708 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:06,708 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:06,709 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:06,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:06,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:06,710 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:06,710 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:06,710 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:06,710 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:06,710 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:06,711 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:06,711 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:06,711 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:06,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:06,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:06,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:06,711 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:06,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:06,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:06,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:06,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:06,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:06,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:06,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:06,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:06,718 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:06,718 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:06,718 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:06,718 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:06,718 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:06,718 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:06,725 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:06,725 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:06,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:06,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:06,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:06,725 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:06,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:06,727 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:06,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:06,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:06,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:06,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:06,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:06,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:06,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:06,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:06,734 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:06,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:06,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:06,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:06,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:06,737 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:06,737 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:06,737 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:06,737 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:06,737 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:06,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:06,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:06,737 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:06,737 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:06,737 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:06,737 DEBUG [Thread-1666] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:06,737 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:06,737 DEBUG [Thread-1666] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:06,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:06,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:06,737 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:06,738 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:06,738 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:06,738 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:06,838 DEBUG [Thread-1666] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:06,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:06,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:06,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:06,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:06,842 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:06,842 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:06,842 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:06,842 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:06,843 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:06,843 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:06,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on 3ab680f7c89c77c9be754a4581df5bc8 2023-03-31 16:55:06,844 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3ab680f7c89c77c9be754a4581df5bc8 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-31 16:55:06,844 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:06,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:06,845 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:06,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:06,845 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=A 2023-03-31 16:55:06,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:06,845 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:06,845 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:06,845 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:06,845 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=B 2023-03-31 16:55:06,846 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:06,846 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=C 2023-03-31 16:55:06,846 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:06,846 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:06,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:06,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:06,846 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:06,846 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:06,847 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:06,847 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:06,847 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:06,847 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:06,856 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:06,856 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:06,856 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:55:06,856 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. started... 2023-03-31 16:55:06,856 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. as already flushing 2023-03-31 16:55:06,856 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:55:06,856 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:06,856 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:06,856 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:06,870 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:06,870 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:06,877 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:06,877 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:06,877 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:06,879 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1263 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/A/77c3709b1c4c4548a55545fb61522094 2023-03-31 16:55:06,879 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:06,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:06,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:06,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:06,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:06,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:06,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:06,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:06,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:06,883 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:06,883 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@323a620[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:06,883 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:06,883 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:06,885 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:06,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:06,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:06,885 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:06,885 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:06,885 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:06,885 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:06,885 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:06,895 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:06,895 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:06,895 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:06,895 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:06,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:06,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:06,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:06,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:06,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:06,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:06,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:06,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:06,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:06,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:06,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:06,898 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:06,898 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:06,898 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:06,898 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:06,898 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:06,898 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:06,900 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:06,900 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:06,900 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:06,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:06,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:06,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:06,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:06,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:06,901 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:06,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:06,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:06,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:06,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:06,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:06,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:06,903 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:06,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:06,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:06,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:06,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:06,907 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:06,907 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:06,907 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:06,907 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:06,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:06,907 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:06,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:06,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:06,907 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:06,908 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:06,908 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:06,908 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:06,908 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:06,908 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:06,907 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:06,908 DEBUG [Thread-1666] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:06,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:06,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:06,908 DEBUG [Thread-1666] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:06,909 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:06,911 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680281706874, max timestamp=1680281706909Num uniques 859; 2023-03-31 16:55:06,912 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680281706874, max timestamp=1680281706909Num uniques 859; 2023-03-31 16:55:06,912 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 16:55:06,912 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 16:55:06,911 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680281706874, max timestamp=1680281706909Num uniques 859; 2023-03-31 16:55:06,912 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 16:55:06,914 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1263 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/B/3b05da6685e14df5afcdfafef2d786ac 2023-03-31 16:55:06,951 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:06,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 394 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40768 deadline: 1680281766948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:06,954 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:06,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 510 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1680281766951, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:06,958 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1263 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/C/b71c651c8d224db2be49420bdda3c856 2023-03-31 16:55:06,966 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/A/77c3709b1c4c4548a55545fb61522094 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/77c3709b1c4c4548a55545fb61522094 2023-03-31 16:55:06,970 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/77c3709b1c4c4548a55545fb61522094, entries=250, sequenceid=1263, filesize=16.7 K 2023-03-31 16:55:06,972 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/B/3b05da6685e14df5afcdfafef2d786ac as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/3b05da6685e14df5afcdfafef2d786ac 2023-03-31 16:55:06,987 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/3b05da6685e14df5afcdfafef2d786ac, entries=150, sequenceid=1263, filesize=11.9 K 2023-03-31 16:55:06,988 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/C/b71c651c8d224db2be49420bdda3c856 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/b71c651c8d224db2be49420bdda3c856 2023-03-31 16:55:06,992 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/b71c651c8d224db2be49420bdda3c856, entries=150, sequenceid=1263, filesize=11.9 K 2023-03-31 16:55:06,993 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 3ab680f7c89c77c9be754a4581df5bc8 in 149ms, sequenceid=1263, compaction requested=true 2023-03-31 16:55:06,994 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3ab680f7c89c77c9be754a4581df5bc8: 2023-03-31 16:55:06,994 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:55:06,994 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:55:06,994 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 16:55:06,994 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 16:55:06,994 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-31 16:55:06,998 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 54850 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 16:55:06,998 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): 3ab680f7c89c77c9be754a4581df5bc8/A is initiating minor compaction (all files) 2023-03-31 16:55:06,998 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3ab680f7c89c77c9be754a4581df5bc8/A in TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:55:06,998 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/b97db0a56b87455eaa3afb8cc2386acf, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/8901902bfc7c4d4986e2b1533fe344aa, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/389d272996f54996a8337f3c06d0bb51, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/77c3709b1c4c4548a55545fb61522094] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp, totalSize=53.6 K 2023-03-31 16:55:06,999 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting b97db0a56b87455eaa3afb8cc2386acf, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=1202, earliestPutTs=1680281705005 2023-03-31 16:55:06,999 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49970 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 16:55:06,999 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1912): 3ab680f7c89c77c9be754a4581df5bc8/B is initiating minor compaction (all files) 2023-03-31 16:55:06,999 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3ab680f7c89c77c9be754a4581df5bc8/B in TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:55:06,999 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/e7adccabb403473fbec840e4e85be90e, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/61f61c30c60c4cef87c35f2cd26d9e11, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/9e77f7fe217f4c3b899fbf8d96a5bdee, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/3b05da6685e14df5afcdfafef2d786ac] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp, totalSize=48.8 K 2023-03-31 16:55:07,000 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting e7adccabb403473fbec840e4e85be90e, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=1202, earliestPutTs=1680281705005 2023-03-31 16:55:07,000 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 8901902bfc7c4d4986e2b1533fe344aa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1238, earliestPutTs=1680281705067 2023-03-31 16:55:07,000 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 61f61c30c60c4cef87c35f2cd26d9e11, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1238, earliestPutTs=1680281705067 2023-03-31 16:55:07,001 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 389d272996f54996a8337f3c06d0bb51, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1252, earliestPutTs=1680281705708 2023-03-31 16:55:07,001 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 9e77f7fe217f4c3b899fbf8d96a5bdee, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1252, earliestPutTs=1680281705708 2023-03-31 16:55:07,002 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 3b05da6685e14df5afcdfafef2d786ac, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1263, earliestPutTs=1680281706835 2023-03-31 16:55:07,002 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 77c3709b1c4c4548a55545fb61522094, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=1263, earliestPutTs=1680281706832 2023-03-31 16:55:07,008 DEBUG [Thread-1666] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:07,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:07,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:07,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:07,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:07,020 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:07,020 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:07,020 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:07,020 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:07,021 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:07,021 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:07,022 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:07,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:07,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:07,022 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:07,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:07,023 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:07,023 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:07,023 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:07,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:07,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:07,023 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:07,030 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): 3ab680f7c89c77c9be754a4581df5bc8#A#compaction#326 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:55:07,041 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:07,041 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:07,041 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:07,041 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:07,041 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:07,050 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3ab680f7c89c77c9be754a4581df5bc8#B#compaction#327 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:55:07,057 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3ab680f7c89c77c9be754a4581df5bc8 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-03-31 16:55:07,057 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=A 2023-03-31 16:55:07,057 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:55:07,057 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=B 2023-03-31 16:55:07,057 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:55:07,057 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=C 2023-03-31 16:55:07,057 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:55:07,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on 3ab680f7c89c77c9be754a4581df5bc8 2023-03-31 16:55:07,064 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:07,064 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:07,068 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:55:07,069 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. started... 2023-03-31 16:55:07,069 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. as already flushing 2023-03-31 16:55:07,069 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:55:07,069 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:07,069 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:07,069 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:07,081 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:07,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 400 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40768 deadline: 1680281767076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:07,082 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:07,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 516 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1680281767077, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:07,084 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:07,084 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:07,085 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:07,086 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:07,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:07,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:07,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:07,086 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:07,086 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:07,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:07,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:07,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:07,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:07,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:07,088 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:07,088 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4fbd3d05[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:07,088 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:07,088 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:07,090 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:07,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:07,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:07,090 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:07,090 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:07,090 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:07,090 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:07,090 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:07,092 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:07,092 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:07,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:07,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:07,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:07,092 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:07,093 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:07,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:07,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:07,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:07,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:07,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:07,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:07,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:07,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:07,095 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:07,095 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:07,095 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:07,095 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:07,095 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:07,095 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:07,096 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:07,096 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:07,097 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:07,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:07,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:07,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:07,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:07,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:07,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:07,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:07,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:07,097 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:07,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:07,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:07,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:07,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:07,098 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:07,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:07,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:07,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:07,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:07,101 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:07,101 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:07,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:07,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:07,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:07,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:07,101 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:07,101 DEBUG [Thread-1666] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:07,101 DEBUG [Thread-1666] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:07,101 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:07,101 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:07,101 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:07,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:07,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:07,101 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:07,101 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:07,102 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:07,102 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:07,102 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:07,184 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:07,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 402 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40768 deadline: 1680281767182, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:07,185 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:07,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 518 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1680281767183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:07,201 DEBUG [Thread-1666] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:07,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:07,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:07,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:07,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:07,205 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:07,205 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:07,205 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:07,205 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:07,206 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:07,206 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:07,207 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:07,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:07,208 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:07,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:07,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:07,208 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:07,208 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:07,209 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:07,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:07,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:07,209 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:07,209 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:07,209 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:07,209 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:07,210 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:07,210 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:07,211 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:07,211 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:07,211 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:55:07,211 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. started... 2023-03-31 16:55:07,211 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. as already flushing 2023-03-31 16:55:07,211 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:55:07,211 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:07,211 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:07,211 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:07,212 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:07,212 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:07,214 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:07,215 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:07,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:07,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:07,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:07,215 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:07,215 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:07,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:07,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:07,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:07,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:07,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:07,218 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:07,218 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49e72a7a[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:07,218 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:07,218 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:07,219 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:07,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:07,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:07,220 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:07,220 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:07,220 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:07,220 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:07,220 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:07,221 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:07,221 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:07,221 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:07,221 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:07,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:07,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:07,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:07,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:07,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:07,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:07,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:07,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:07,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:07,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:07,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:07,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:07,224 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:07,224 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:07,224 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:07,224 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:07,224 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:07,225 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:07,225 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:07,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:07,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:07,225 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:07,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:07,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:07,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:07,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:07,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:07,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:07,226 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:07,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:07,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:07,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:07,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:07,227 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:07,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:07,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:07,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:07,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:07,231 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:07,231 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:07,231 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:07,231 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:07,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:07,231 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:07,231 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:07,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:07,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:07,232 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:07,231 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:07,231 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:07,232 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:07,232 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:07,232 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:07,232 DEBUG [Thread-1666] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:07,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:07,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:07,232 DEBUG [Thread-1666] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:07,332 DEBUG [Thread-1666] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:07,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:07,334 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:07,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:07,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:07,335 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:07,335 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:07,335 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:07,335 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:07,336 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:07,336 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:07,337 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:07,337 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:07,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:07,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:07,337 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:07,337 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:07,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:07,338 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:07,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:07,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:07,338 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:07,338 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:07,338 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:07,338 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:07,338 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:07,338 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:07,339 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:07,339 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:07,339 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:55:07,339 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. started... 2023-03-31 16:55:07,339 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. as already flushing 2023-03-31 16:55:07,339 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:55:07,339 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:07,339 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:07,339 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:07,339 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:07,339 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:07,340 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:07,340 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:07,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:07,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:07,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:07,340 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:07,340 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:07,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:07,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:07,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:07,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:07,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:07,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:07,342 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:07,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@407de60[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:07,342 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:07,351 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:07,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:07,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:07,351 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:07,351 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:07,351 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:07,351 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:07,351 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:07,352 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:07,352 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:07,352 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:07,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:07,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:07,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:07,352 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:07,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:07,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:07,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:07,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:07,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:07,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:07,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:07,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:07,355 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:07,355 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:07,355 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:07,355 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:07,355 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:07,355 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:07,356 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:07,356 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:07,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:07,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:07,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:07,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:07,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:07,356 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:07,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:07,357 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:07,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:07,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:07,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:07,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:07,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:07,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:07,358 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:07,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:07,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:07,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:07,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:07,360 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:07,360 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:07,360 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:07,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:07,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:07,360 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:07,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:07,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:07,361 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:07,361 DEBUG [Thread-1666] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:07,360 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:07,361 DEBUG [Thread-1666] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:07,360 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:07,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:07,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:07,361 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:07,361 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:07,361 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:07,361 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:07,362 DEBUG [Thread-1667] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2ce5573e to 127.0.0.1:51916 2023-03-31 16:55:07,362 DEBUG [Thread-1667] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 16:55:07,363 DEBUG [Thread-1675] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6fcbaba8 to 127.0.0.1:51916 2023-03-31 16:55:07,363 DEBUG [Thread-1675] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 16:55:07,364 DEBUG [Thread-1671] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x12115052 to 127.0.0.1:51916 2023-03-31 16:55:07,364 DEBUG [Thread-1671] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 16:55:07,365 DEBUG [Thread-1669] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x087b3095 to 127.0.0.1:51916 2023-03-31 16:55:07,365 DEBUG [Thread-1669] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 16:55:07,366 DEBUG [Thread-1673] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7e65578e to 127.0.0.1:51916 2023-03-31 16:55:07,366 DEBUG [Thread-1673] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 16:55:07,385 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:07,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 404 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40768 deadline: 1680281767385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:07,386 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:07,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 520 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1680281767386, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:07,461 DEBUG [Thread-1666] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:07,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:07,490 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=1300 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/A/1059fc7e62a24478a048c7e2537c10f9 2023-03-31 16:55:07,490 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/A/af4b84597c784336816cb6830cc3703b as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/af4b84597c784336816cb6830cc3703b 2023-03-31 16:55:07,491 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/B/574be8c010074f86a7b8981c1180cdf8 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/574be8c010074f86a7b8981c1180cdf8 2023-03-31 16:55:07,494 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3ab680f7c89c77c9be754a4581df5bc8/A of 3ab680f7c89c77c9be754a4581df5bc8 into af4b84597c784336816cb6830cc3703b(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:55:07,494 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3ab680f7c89c77c9be754a4581df5bc8: 2023-03-31 16:55:07,494 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8., storeName=3ab680f7c89c77c9be754a4581df5bc8/A, priority=12, startTime=1680281706994; duration=0sec 2023-03-31 16:55:07,495 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:55:07,495 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3ab680f7c89c77c9be754a4581df5bc8/B of 3ab680f7c89c77c9be754a4581df5bc8 into 574be8c010074f86a7b8981c1180cdf8(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:55:07,495 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 16:55:07,495 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3ab680f7c89c77c9be754a4581df5bc8: 2023-03-31 16:55:07,495 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8., storeName=3ab680f7c89c77c9be754a4581df5bc8/B, priority=12, startTime=1680281706994; duration=0sec 2023-03-31 16:55:07,495 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:55:07,496 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49970 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 16:55:07,496 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): 3ab680f7c89c77c9be754a4581df5bc8/C is initiating minor compaction (all files) 2023-03-31 16:55:07,496 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3ab680f7c89c77c9be754a4581df5bc8/C in TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:55:07,496 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/7eead4bbc30548fba31a7624f1c1eac7, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/fdae192f025d4439822ab420380c3884, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/673908c6b2c9419c8b58f5e16d13ef53, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/b71c651c8d224db2be49420bdda3c856] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp, totalSize=48.8 K 2023-03-31 16:55:07,496 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 7eead4bbc30548fba31a7624f1c1eac7, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=1202, earliestPutTs=1680281705005 2023-03-31 16:55:07,497 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting fdae192f025d4439822ab420380c3884, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1238, earliestPutTs=1680281705067 2023-03-31 16:55:07,497 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 673908c6b2c9419c8b58f5e16d13ef53, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1252, earliestPutTs=1680281705708 2023-03-31 16:55:07,497 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting b71c651c8d224db2be49420bdda3c856, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1263, earliestPutTs=1680281706835 2023-03-31 16:55:07,503 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): 3ab680f7c89c77c9be754a4581df5bc8#C#compaction#330 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:55:07,687 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:07,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 406 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40768 deadline: 1680281767687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:07,688 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:07,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 522 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1680281767688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:07,899 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=1300 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/B/f70cf9e46cab42e283d182e9f9a3c1a5 2023-03-31 16:55:07,909 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/C/422bc5ea62bb44c08538569684537aab as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/422bc5ea62bb44c08538569684537aab 2023-03-31 16:55:07,912 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3ab680f7c89c77c9be754a4581df5bc8/C of 3ab680f7c89c77c9be754a4581df5bc8 into 422bc5ea62bb44c08538569684537aab(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:55:07,912 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3ab680f7c89c77c9be754a4581df5bc8: 2023-03-31 16:55:07,912 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8., storeName=3ab680f7c89c77c9be754a4581df5bc8/C, priority=12, startTime=1680281706994; duration=0sec 2023-03-31 16:55:07,912 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:55:08,190 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:08,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 408 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40768 deadline: 1680281768189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:08,190 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:08,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 524 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1680281768190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:08,307 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=1300 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/C/96b38dff61494cdda58d1cd885227261 2023-03-31 16:55:08,311 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/A/1059fc7e62a24478a048c7e2537c10f9 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/1059fc7e62a24478a048c7e2537c10f9 2023-03-31 16:55:08,314 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/1059fc7e62a24478a048c7e2537c10f9, entries=200, sequenceid=1300, filesize=14.3 K 2023-03-31 16:55:08,315 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/B/f70cf9e46cab42e283d182e9f9a3c1a5 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/f70cf9e46cab42e283d182e9f9a3c1a5 2023-03-31 16:55:08,318 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/f70cf9e46cab42e283d182e9f9a3c1a5, entries=150, sequenceid=1300, filesize=11.9 K 2023-03-31 16:55:08,318 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/C/96b38dff61494cdda58d1cd885227261 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/96b38dff61494cdda58d1cd885227261 2023-03-31 16:55:08,322 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/96b38dff61494cdda58d1cd885227261, entries=150, sequenceid=1300, filesize=11.9 K 2023-03-31 16:55:08,322 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 3ab680f7c89c77c9be754a4581df5bc8 in 1265ms, sequenceid=1300, compaction requested=false 2023-03-31 16:55:08,322 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3ab680f7c89c77c9be754a4581df5bc8: 2023-03-31 16:55:09,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on 3ab680f7c89c77c9be754a4581df5bc8 2023-03-31 16:55:09,194 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3ab680f7c89c77c9be754a4581df5bc8 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-31 16:55:09,194 DEBUG [Thread-1658] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x00ca898c to 127.0.0.1:51916 2023-03-31 16:55:09,194 DEBUG [Thread-1658] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 16:55:09,195 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=A 2023-03-31 16:55:09,195 DEBUG [Thread-1656] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x262f4570 to 127.0.0.1:51916 2023-03-31 16:55:09,195 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:09,195 DEBUG [Thread-1656] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 16:55:09,195 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=B 2023-03-31 16:55:09,195 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:09,195 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=C 2023-03-31 16:55:09,195 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:09,601 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=1315 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/A/e1d3b0bd4afa422e94a5c6d273b661b2 2023-03-31 16:55:10,012 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=1315 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/B/c282ae5d607e43baae8fdb9391ac5c10 2023-03-31 16:55:10,021 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=1315 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/C/4246b31358fe460daea429a11e2bd7a0 2023-03-31 16:55:10,025 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/A/e1d3b0bd4afa422e94a5c6d273b661b2 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/e1d3b0bd4afa422e94a5c6d273b661b2 2023-03-31 16:55:10,029 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/e1d3b0bd4afa422e94a5c6d273b661b2, entries=150, sequenceid=1315, filesize=11.9 K 2023-03-31 16:55:10,030 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/B/c282ae5d607e43baae8fdb9391ac5c10 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/c282ae5d607e43baae8fdb9391ac5c10 2023-03-31 16:55:10,034 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/c282ae5d607e43baae8fdb9391ac5c10, entries=150, sequenceid=1315, filesize=11.9 K 2023-03-31 16:55:10,035 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/C/4246b31358fe460daea429a11e2bd7a0 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/4246b31358fe460daea429a11e2bd7a0 2023-03-31 16:55:10,039 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/4246b31358fe460daea429a11e2bd7a0, entries=150, sequenceid=1315, filesize=11.9 K 2023-03-31 16:55:10,039 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=0 B/0 for 3ab680f7c89c77c9be754a4581df5bc8 in 845ms, sequenceid=1315, compaction requested=true 2023-03-31 16:55:10,039 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3ab680f7c89c77c9be754a4581df5bc8: 2023-03-31 16:55:10,039 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:55:10,039 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 16:55:10,040 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 16:55:10,039 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:55:10,040 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:55:10,040 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40360 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 16:55:10,041 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37920 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 16:55:10,041 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1912): 3ab680f7c89c77c9be754a4581df5bc8/A is initiating minor compaction (all files) 2023-03-31 16:55:10,041 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3ab680f7c89c77c9be754a4581df5bc8/A in TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:55:10,041 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/af4b84597c784336816cb6830cc3703b, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/1059fc7e62a24478a048c7e2537c10f9, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/e1d3b0bd4afa422e94a5c6d273b661b2] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp, totalSize=39.4 K 2023-03-31 16:55:10,041 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): 3ab680f7c89c77c9be754a4581df5bc8/B is initiating minor compaction (all files) 2023-03-31 16:55:10,041 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3ab680f7c89c77c9be754a4581df5bc8/B in TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:55:10,041 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/574be8c010074f86a7b8981c1180cdf8, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/f70cf9e46cab42e283d182e9f9a3c1a5, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/c282ae5d607e43baae8fdb9391ac5c10] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp, totalSize=37.0 K 2023-03-31 16:55:10,041 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting af4b84597c784336816cb6830cc3703b, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=1263, earliestPutTs=1680281706835 2023-03-31 16:55:10,041 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 574be8c010074f86a7b8981c1180cdf8, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=1263, earliestPutTs=1680281706835 2023-03-31 16:55:10,041 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 1059fc7e62a24478a048c7e2537c10f9, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=1300, earliestPutTs=1680281706939 2023-03-31 16:55:10,042 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting f70cf9e46cab42e283d182e9f9a3c1a5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1300, earliestPutTs=1680281706939 2023-03-31 16:55:10,042 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting e1d3b0bd4afa422e94a5c6d273b661b2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1315, earliestPutTs=1680281707075 2023-03-31 16:55:10,042 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting c282ae5d607e43baae8fdb9391ac5c10, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1315, earliestPutTs=1680281707075 2023-03-31 16:55:10,049 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): 3ab680f7c89c77c9be754a4581df5bc8#B#compaction#335 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:55:10,057 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3ab680f7c89c77c9be754a4581df5bc8#A#compaction#336 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:55:10,064 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/B/e7b3c8e310e24e11a751f2d474d9d392 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/e7b3c8e310e24e11a751f2d474d9d392 2023-03-31 16:55:10,068 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3ab680f7c89c77c9be754a4581df5bc8/B of 3ab680f7c89c77c9be754a4581df5bc8 into e7b3c8e310e24e11a751f2d474d9d392(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:55:10,068 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3ab680f7c89c77c9be754a4581df5bc8: 2023-03-31 16:55:10,068 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8., storeName=3ab680f7c89c77c9be754a4581df5bc8/B, priority=13, startTime=1680281710039; duration=0sec 2023-03-31 16:55:10,068 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:55:10,068 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 16:55:10,069 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37920 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 16:55:10,069 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): 3ab680f7c89c77c9be754a4581df5bc8/C is initiating minor compaction (all files) 2023-03-31 16:55:10,069 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3ab680f7c89c77c9be754a4581df5bc8/C in TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:55:10,070 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/422bc5ea62bb44c08538569684537aab, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/96b38dff61494cdda58d1cd885227261, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/4246b31358fe460daea429a11e2bd7a0] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp, totalSize=37.0 K 2023-03-31 16:55:10,070 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 422bc5ea62bb44c08538569684537aab, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=1263, earliestPutTs=1680281706835 2023-03-31 16:55:10,070 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 96b38dff61494cdda58d1cd885227261, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1300, earliestPutTs=1680281706939 2023-03-31 16:55:10,071 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 4246b31358fe460daea429a11e2bd7a0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1315, earliestPutTs=1680281707075 2023-03-31 16:55:10,077 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): 3ab680f7c89c77c9be754a4581df5bc8#C#compaction#337 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:55:10,473 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/A/bcdba742fa9f4f68b5d9dadaa0b10ef9 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/bcdba742fa9f4f68b5d9dadaa0b10ef9 2023-03-31 16:55:10,476 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3ab680f7c89c77c9be754a4581df5bc8/A of 3ab680f7c89c77c9be754a4581df5bc8 into bcdba742fa9f4f68b5d9dadaa0b10ef9(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:55:10,476 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3ab680f7c89c77c9be754a4581df5bc8: 2023-03-31 16:55:10,476 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8., storeName=3ab680f7c89c77c9be754a4581df5bc8/A, priority=13, startTime=1680281710039; duration=0sec 2023-03-31 16:55:10,477 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:55:10,484 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/C/2146702411424b0e907d889c6c025a8a as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/2146702411424b0e907d889c6c025a8a 2023-03-31 16:55:10,487 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3ab680f7c89c77c9be754a4581df5bc8/C of 3ab680f7c89c77c9be754a4581df5bc8 into 2146702411424b0e907d889c6c025a8a(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:55:10,487 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3ab680f7c89c77c9be754a4581df5bc8: 2023-03-31 16:55:10,487 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8., storeName=3ab680f7c89c77c9be754a4581df5bc8/C, priority=13, startTime=1680281710040; duration=0sec 2023-03-31 16:55:10,487 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:55:14,060 DEBUG [Thread-1662] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7480f9da to 127.0.0.1:51916 2023-03-31 16:55:14,060 DEBUG [Thread-1662] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 16:55:14,065 DEBUG [Thread-1664] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1a24d617 to 127.0.0.1:51916 2023-03-31 16:55:14,065 DEBUG [Thread-1664] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 16:55:14,119 DEBUG [Thread-1660] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6585a70a to 127.0.0.1:51916 2023-03-31 16:55:14,119 DEBUG [Thread-1660] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 16:55:14,119 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-31 16:55:14,119 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(394): wrote 279 2023-03-31 16:55:14,119 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(394): wrote 393 2023-03-31 16:55:14,119 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(394): wrote 103 2023-03-31 16:55:14,119 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(394): wrote 253 2023-03-31 16:55:14,119 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(394): wrote 157 2023-03-31 16:55:14,119 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-31 16:55:14,119 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(398): read 5125 2023-03-31 16:55:14,119 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(398): read 5069 2023-03-31 16:55:14,119 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(398): read 4965 2023-03-31 16:55:14,119 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(398): read 5118 2023-03-31 16:55:14,119 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(398): read 5082 2023-03-31 16:55:14,119 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-31 16:55:14,119 INFO [Listener at localhost/44135] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-31 16:55:14,119 DEBUG [Listener at localhost/44135] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x22b4eae5 to 127.0.0.1:51916 2023-03-31 16:55:14,120 DEBUG [Listener at localhost/44135] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 16:55:14,120 INFO [Listener at localhost/44135] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-31 16:55:14,121 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-03-31 16:55:14,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure2.ProcedureExecutor(1029): Stored pid=31, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-31 16:55:14,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-31 16:55:14,124 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680281714124"}]},"ts":"1680281714124"} 2023-03-31 16:55:14,124 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-31 16:55:14,126 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-31 16:55:14,126 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=3ab680f7c89c77c9be754a4581df5bc8, UNASSIGN}] 2023-03-31 16:55:14,127 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=3ab680f7c89c77c9be754a4581df5bc8, UNASSIGN 2023-03-31 16:55:14,128 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=3ab680f7c89c77c9be754a4581df5bc8, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:14,128 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680281714128"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680281714128"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680281714128"}]},"ts":"1680281714128"} 2023-03-31 16:55:14,129 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=32, state=RUNNABLE; CloseRegionProcedure 3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375}] 2023-03-31 16:55:14,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-31 16:55:14,280 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 3ab680f7c89c77c9be754a4581df5bc8 2023-03-31 16:55:14,281 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 3ab680f7c89c77c9be754a4581df5bc8, disabling compactions & flushes 2023-03-31 16:55:14,281 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:55:14,281 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:55:14,281 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. after waiting 0 ms 2023-03-31 16:55:14,281 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:55:14,281 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 3ab680f7c89c77c9be754a4581df5bc8 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-03-31 16:55:14,281 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=A 2023-03-31 16:55:14,281 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:14,281 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=B 2023-03-31 16:55:14,281 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:14,281 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ab680f7c89c77c9be754a4581df5bc8, store=C 2023-03-31 16:55:14,281 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:14,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-31 16:55:14,687 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=1324 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/A/e28e1c5636514d3290f318a6458b78d1 2023-03-31 16:55:14,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-31 16:55:15,096 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=1324 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/B/c515673d6b994d9ca27697afac8f2b97 2023-03-31 16:55:15,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-31 16:55:15,506 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=1324 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/C/5de8de14f82c498fb28160bbf6901c45 2023-03-31 16:55:15,510 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/A/e28e1c5636514d3290f318a6458b78d1 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/e28e1c5636514d3290f318a6458b78d1 2023-03-31 16:55:15,513 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/e28e1c5636514d3290f318a6458b78d1, entries=100, sequenceid=1324, filesize=9.5 K 2023-03-31 16:55:15,514 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/B/c515673d6b994d9ca27697afac8f2b97 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/c515673d6b994d9ca27697afac8f2b97 2023-03-31 16:55:15,518 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/c515673d6b994d9ca27697afac8f2b97, entries=100, sequenceid=1324, filesize=9.5 K 2023-03-31 16:55:15,518 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/.tmp/C/5de8de14f82c498fb28160bbf6901c45 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/5de8de14f82c498fb28160bbf6901c45 2023-03-31 16:55:15,521 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/5de8de14f82c498fb28160bbf6901c45, entries=100, sequenceid=1324, filesize=9.5 K 2023-03-31 16:55:15,522 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=0 B/0 for 3ab680f7c89c77c9be754a4581df5bc8 in 1241ms, sequenceid=1324, compaction requested=false 2023-03-31 16:55:15,545 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/c682f0805d12414c8ec7285708401af1, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/0c73b5d47d0645778db68c0c3a116615, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/dd5f876a01a84c578e5f6ad2b532086f, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/4a936689ba9243df8d88098f23782fce, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/971dc5a195be4ed8a2f7cb274db0afae, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/dfc18ad9987c40bebca50113a82365fd, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/6e2435bd4c00454db912d5779bcdb3de, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/635d2c14218f4906bfe16bb619d60aa8, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/e35bb37161cb489ba3eea7d527b760fe, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/654233eeecf943a4bb2d01ba2861d573, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/613a872127fb4c298ce182599b1d6a61, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/7a38f1c9abf54c96b1180fda896def39, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/2c2aee8a7bdf467789516978cd0be6f7, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/fc90c4f61e574f18b3e65b94d6e7ed1d, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/3e4e7da004124a1f8d26fc603c78d4be, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/e8b5c731ec8c4e0e8fda7b5011d25c06, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/09ebd0ffb0944b459a2f9da6ff0bdd11, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/045570ed7d1c4fb9b03789a80f2ceb12, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/836b7b6f46db48a99ba766ca44b546d7, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/34d9013ab93f484b96736f66a0b54e61, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/beae9347000340bda38546557d5fc966, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/5d0f854716dc4ae18057d463657f320c, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/2aaf8e20f90c4f62bba9a02600310c8d, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/1d3136f776ba45b29debf469425c5075, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/8c93aca3f27a44b78d84cc1fb41ddc3e, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/1551de7b75f94c51904f23a8e83075a6, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/99a192203327413cb219ad6690710663, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/e5baf1c23f6f44aa8ce99062f35825c5, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/b0d180f5c10242389f19ac6575f710e6, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/7c8732d0582f46abbad4387b9e99b01d, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/b2c69c42d64a48d28f15d77bb102a09d, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/58dd9207b4844db7b35bb2979cd44063, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/bc8dad9976d441f898d5c55088c585e1, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/87dab3be29014a899f4cec085226ed44, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/9748660522264f609f438d3b20078eb6, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/b97db0a56b87455eaa3afb8cc2386acf, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/ef607b1205a24a01b772e99b884ce549, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/8901902bfc7c4d4986e2b1533fe344aa, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/389d272996f54996a8337f3c06d0bb51, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/77c3709b1c4c4548a55545fb61522094, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/af4b84597c784336816cb6830cc3703b, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/1059fc7e62a24478a048c7e2537c10f9, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/e1d3b0bd4afa422e94a5c6d273b661b2] to archive 2023-03-31 16:55:15,546 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-31 16:55:15,547 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/c682f0805d12414c8ec7285708401af1 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/c682f0805d12414c8ec7285708401af1 2023-03-31 16:55:15,548 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/0c73b5d47d0645778db68c0c3a116615 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/0c73b5d47d0645778db68c0c3a116615 2023-03-31 16:55:15,549 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/dd5f876a01a84c578e5f6ad2b532086f to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/dd5f876a01a84c578e5f6ad2b532086f 2023-03-31 16:55:15,550 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/4a936689ba9243df8d88098f23782fce to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/4a936689ba9243df8d88098f23782fce 2023-03-31 16:55:15,551 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/971dc5a195be4ed8a2f7cb274db0afae to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/971dc5a195be4ed8a2f7cb274db0afae 2023-03-31 16:55:15,551 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/dfc18ad9987c40bebca50113a82365fd to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/dfc18ad9987c40bebca50113a82365fd 2023-03-31 16:55:15,552 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/6e2435bd4c00454db912d5779bcdb3de to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/6e2435bd4c00454db912d5779bcdb3de 2023-03-31 16:55:15,553 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/635d2c14218f4906bfe16bb619d60aa8 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/635d2c14218f4906bfe16bb619d60aa8 2023-03-31 16:55:15,554 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/e35bb37161cb489ba3eea7d527b760fe to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/e35bb37161cb489ba3eea7d527b760fe 2023-03-31 16:55:15,555 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/654233eeecf943a4bb2d01ba2861d573 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/654233eeecf943a4bb2d01ba2861d573 2023-03-31 16:55:15,556 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/613a872127fb4c298ce182599b1d6a61 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/613a872127fb4c298ce182599b1d6a61 2023-03-31 16:55:15,557 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/7a38f1c9abf54c96b1180fda896def39 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/7a38f1c9abf54c96b1180fda896def39 2023-03-31 16:55:15,558 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/2c2aee8a7bdf467789516978cd0be6f7 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/2c2aee8a7bdf467789516978cd0be6f7 2023-03-31 16:55:15,558 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/fc90c4f61e574f18b3e65b94d6e7ed1d to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/fc90c4f61e574f18b3e65b94d6e7ed1d 2023-03-31 16:55:15,559 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/3e4e7da004124a1f8d26fc603c78d4be to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/3e4e7da004124a1f8d26fc603c78d4be 2023-03-31 16:55:15,560 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/e8b5c731ec8c4e0e8fda7b5011d25c06 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/e8b5c731ec8c4e0e8fda7b5011d25c06 2023-03-31 16:55:15,561 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/09ebd0ffb0944b459a2f9da6ff0bdd11 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/09ebd0ffb0944b459a2f9da6ff0bdd11 2023-03-31 16:55:15,562 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/045570ed7d1c4fb9b03789a80f2ceb12 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/045570ed7d1c4fb9b03789a80f2ceb12 2023-03-31 16:55:15,563 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/836b7b6f46db48a99ba766ca44b546d7 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/836b7b6f46db48a99ba766ca44b546d7 2023-03-31 16:55:15,564 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/34d9013ab93f484b96736f66a0b54e61 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/34d9013ab93f484b96736f66a0b54e61 2023-03-31 16:55:15,565 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/beae9347000340bda38546557d5fc966 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/beae9347000340bda38546557d5fc966 2023-03-31 16:55:15,566 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/5d0f854716dc4ae18057d463657f320c to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/5d0f854716dc4ae18057d463657f320c 2023-03-31 16:55:15,567 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/2aaf8e20f90c4f62bba9a02600310c8d to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/2aaf8e20f90c4f62bba9a02600310c8d 2023-03-31 16:55:15,568 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/1d3136f776ba45b29debf469425c5075 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/1d3136f776ba45b29debf469425c5075 2023-03-31 16:55:15,569 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/8c93aca3f27a44b78d84cc1fb41ddc3e to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/8c93aca3f27a44b78d84cc1fb41ddc3e 2023-03-31 16:55:15,570 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/1551de7b75f94c51904f23a8e83075a6 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/1551de7b75f94c51904f23a8e83075a6 2023-03-31 16:55:15,571 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/99a192203327413cb219ad6690710663 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/99a192203327413cb219ad6690710663 2023-03-31 16:55:15,572 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/e5baf1c23f6f44aa8ce99062f35825c5 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/e5baf1c23f6f44aa8ce99062f35825c5 2023-03-31 16:55:15,573 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/b0d180f5c10242389f19ac6575f710e6 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/b0d180f5c10242389f19ac6575f710e6 2023-03-31 16:55:15,574 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/7c8732d0582f46abbad4387b9e99b01d to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/7c8732d0582f46abbad4387b9e99b01d 2023-03-31 16:55:15,575 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/b2c69c42d64a48d28f15d77bb102a09d to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/b2c69c42d64a48d28f15d77bb102a09d 2023-03-31 16:55:15,576 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/58dd9207b4844db7b35bb2979cd44063 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/58dd9207b4844db7b35bb2979cd44063 2023-03-31 16:55:15,577 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/bc8dad9976d441f898d5c55088c585e1 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/bc8dad9976d441f898d5c55088c585e1 2023-03-31 16:55:15,578 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/87dab3be29014a899f4cec085226ed44 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/87dab3be29014a899f4cec085226ed44 2023-03-31 16:55:15,579 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/9748660522264f609f438d3b20078eb6 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/9748660522264f609f438d3b20078eb6 2023-03-31 16:55:15,580 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/b97db0a56b87455eaa3afb8cc2386acf to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/b97db0a56b87455eaa3afb8cc2386acf 2023-03-31 16:55:15,581 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/ef607b1205a24a01b772e99b884ce549 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/ef607b1205a24a01b772e99b884ce549 2023-03-31 16:55:15,581 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/8901902bfc7c4d4986e2b1533fe344aa to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/8901902bfc7c4d4986e2b1533fe344aa 2023-03-31 16:55:15,582 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/389d272996f54996a8337f3c06d0bb51 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/389d272996f54996a8337f3c06d0bb51 2023-03-31 16:55:15,583 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/77c3709b1c4c4548a55545fb61522094 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/77c3709b1c4c4548a55545fb61522094 2023-03-31 16:55:15,584 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/af4b84597c784336816cb6830cc3703b to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/af4b84597c784336816cb6830cc3703b 2023-03-31 16:55:15,585 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/1059fc7e62a24478a048c7e2537c10f9 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/1059fc7e62a24478a048c7e2537c10f9 2023-03-31 16:55:15,586 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/e1d3b0bd4afa422e94a5c6d273b661b2 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/e1d3b0bd4afa422e94a5c6d273b661b2 2023-03-31 16:55:15,589 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/892607cbf9fb41b8b3c8688d5135ff94, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/3625c7f038f64ec6a85f794a2e6e0f84, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/71e97fca239244f0a79a062450a36929, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/4005fb371227416ab321e650f03677de, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/dd7de93a1c9844dfbd827fd4348d72b0, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/e997c7a82ddb4f5ea3121e9784db51ab, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/bed3473d9b2046fb94a33c30ec707294, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/9717f3a6d39e4fc1975721d3169d0ca3, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/bd9851dfc85a48a7b39c35ae34a368a1, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/12ea4e7eff5d4b728065004a29cfc554, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/50886de2d5214df1a70385edb0901e00, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/e3bfb109cc3e413d865bc65e82158419, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/fd064160575342d1a44f215a6e5cd621, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/ca44fadb4f7d4ee0b171ca6a1d9c8035, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/9dc4c5992230473f9d49755e3bf97dd6, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/296d22848fd14345bf379a7fe7073ab6, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/ab7aea1077564a9099510a0b1276f101, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/00bfb127153c44688c4948bba3ee7ba7, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/aab07af971d84b3a97dc55508856235e, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/f9f0093211874ef0bde6f36d400a105d, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/6f5a347face5438c99636caf356655d8, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/0e09b30a3f844ca289d0053a1aa51691, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/b71d259ddca44b8dbd2c7de5c92ee511, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/5ddd24c9f83249ceb6b51f9a7b6bfd98, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/5d74219bd1e24aa78410f695da469916, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/522dc1b1091d4e3e81c9acb1b4d29446, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/bc97c36c323a4a7cb253bfa2a7557fd8, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/f31076716f5b44e98a2f85982a6bf846, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/f1aacfb5e3b44e6284af625b43b87646, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/bda9a8d5b4134a5ba304f6ce93b0c5df, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/250ebfce1e274dbb9be3ef5ad7fea2de, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/039c3c4caffb4a89b79d6d9a8e399fab, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/3f1db28f389f4843b1b3e2372305da7f, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/d090a5d0cc9141c4988b879f30c96556, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/c2c2950e2772476fb3c62e7477c5cfb6, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/e7adccabb403473fbec840e4e85be90e, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/76b6126c1fb8460aafee4c1bc1fca2ad, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/61f61c30c60c4cef87c35f2cd26d9e11, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/9e77f7fe217f4c3b899fbf8d96a5bdee, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/574be8c010074f86a7b8981c1180cdf8, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/3b05da6685e14df5afcdfafef2d786ac, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/f70cf9e46cab42e283d182e9f9a3c1a5, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/c282ae5d607e43baae8fdb9391ac5c10] to archive 2023-03-31 16:55:15,590 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-31 16:55:15,593 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/892607cbf9fb41b8b3c8688d5135ff94 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/892607cbf9fb41b8b3c8688d5135ff94 2023-03-31 16:55:15,594 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/3625c7f038f64ec6a85f794a2e6e0f84 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/3625c7f038f64ec6a85f794a2e6e0f84 2023-03-31 16:55:15,596 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/71e97fca239244f0a79a062450a36929 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/71e97fca239244f0a79a062450a36929 2023-03-31 16:55:15,597 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/4005fb371227416ab321e650f03677de to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/4005fb371227416ab321e650f03677de 2023-03-31 16:55:15,598 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/dd7de93a1c9844dfbd827fd4348d72b0 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/dd7de93a1c9844dfbd827fd4348d72b0 2023-03-31 16:55:15,599 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/e997c7a82ddb4f5ea3121e9784db51ab to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/e997c7a82ddb4f5ea3121e9784db51ab 2023-03-31 16:55:15,600 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/bed3473d9b2046fb94a33c30ec707294 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/bed3473d9b2046fb94a33c30ec707294 2023-03-31 16:55:15,602 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/9717f3a6d39e4fc1975721d3169d0ca3 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/9717f3a6d39e4fc1975721d3169d0ca3 2023-03-31 16:55:15,603 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/bd9851dfc85a48a7b39c35ae34a368a1 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/bd9851dfc85a48a7b39c35ae34a368a1 2023-03-31 16:55:15,604 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/12ea4e7eff5d4b728065004a29cfc554 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/12ea4e7eff5d4b728065004a29cfc554 2023-03-31 16:55:15,605 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/50886de2d5214df1a70385edb0901e00 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/50886de2d5214df1a70385edb0901e00 2023-03-31 16:55:15,606 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/e3bfb109cc3e413d865bc65e82158419 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/e3bfb109cc3e413d865bc65e82158419 2023-03-31 16:55:15,607 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/fd064160575342d1a44f215a6e5cd621 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/fd064160575342d1a44f215a6e5cd621 2023-03-31 16:55:15,608 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/ca44fadb4f7d4ee0b171ca6a1d9c8035 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/ca44fadb4f7d4ee0b171ca6a1d9c8035 2023-03-31 16:55:15,609 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/9dc4c5992230473f9d49755e3bf97dd6 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/9dc4c5992230473f9d49755e3bf97dd6 2023-03-31 16:55:15,610 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/296d22848fd14345bf379a7fe7073ab6 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/296d22848fd14345bf379a7fe7073ab6 2023-03-31 16:55:15,611 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/ab7aea1077564a9099510a0b1276f101 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/ab7aea1077564a9099510a0b1276f101 2023-03-31 16:55:15,612 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/00bfb127153c44688c4948bba3ee7ba7 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/00bfb127153c44688c4948bba3ee7ba7 2023-03-31 16:55:15,612 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/aab07af971d84b3a97dc55508856235e to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/aab07af971d84b3a97dc55508856235e 2023-03-31 16:55:15,614 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/f9f0093211874ef0bde6f36d400a105d to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/f9f0093211874ef0bde6f36d400a105d 2023-03-31 16:55:15,614 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/6f5a347face5438c99636caf356655d8 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/6f5a347face5438c99636caf356655d8 2023-03-31 16:55:15,615 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/0e09b30a3f844ca289d0053a1aa51691 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/0e09b30a3f844ca289d0053a1aa51691 2023-03-31 16:55:15,616 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/b71d259ddca44b8dbd2c7de5c92ee511 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/b71d259ddca44b8dbd2c7de5c92ee511 2023-03-31 16:55:15,617 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/5ddd24c9f83249ceb6b51f9a7b6bfd98 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/5ddd24c9f83249ceb6b51f9a7b6bfd98 2023-03-31 16:55:15,618 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/5d74219bd1e24aa78410f695da469916 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/5d74219bd1e24aa78410f695da469916 2023-03-31 16:55:15,618 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/522dc1b1091d4e3e81c9acb1b4d29446 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/522dc1b1091d4e3e81c9acb1b4d29446 2023-03-31 16:55:15,619 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/bc97c36c323a4a7cb253bfa2a7557fd8 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/bc97c36c323a4a7cb253bfa2a7557fd8 2023-03-31 16:55:15,620 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/f31076716f5b44e98a2f85982a6bf846 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/f31076716f5b44e98a2f85982a6bf846 2023-03-31 16:55:15,621 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/f1aacfb5e3b44e6284af625b43b87646 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/f1aacfb5e3b44e6284af625b43b87646 2023-03-31 16:55:15,621 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/bda9a8d5b4134a5ba304f6ce93b0c5df to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/bda9a8d5b4134a5ba304f6ce93b0c5df 2023-03-31 16:55:15,622 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/250ebfce1e274dbb9be3ef5ad7fea2de to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/250ebfce1e274dbb9be3ef5ad7fea2de 2023-03-31 16:55:15,623 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/039c3c4caffb4a89b79d6d9a8e399fab to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/039c3c4caffb4a89b79d6d9a8e399fab 2023-03-31 16:55:15,624 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/3f1db28f389f4843b1b3e2372305da7f to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/3f1db28f389f4843b1b3e2372305da7f 2023-03-31 16:55:15,624 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/d090a5d0cc9141c4988b879f30c96556 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/d090a5d0cc9141c4988b879f30c96556 2023-03-31 16:55:15,625 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/c2c2950e2772476fb3c62e7477c5cfb6 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/c2c2950e2772476fb3c62e7477c5cfb6 2023-03-31 16:55:15,626 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/e7adccabb403473fbec840e4e85be90e to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/e7adccabb403473fbec840e4e85be90e 2023-03-31 16:55:15,627 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/76b6126c1fb8460aafee4c1bc1fca2ad to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/76b6126c1fb8460aafee4c1bc1fca2ad 2023-03-31 16:55:15,628 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/61f61c30c60c4cef87c35f2cd26d9e11 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/61f61c30c60c4cef87c35f2cd26d9e11 2023-03-31 16:55:15,628 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/9e77f7fe217f4c3b899fbf8d96a5bdee to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/9e77f7fe217f4c3b899fbf8d96a5bdee 2023-03-31 16:55:15,629 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/574be8c010074f86a7b8981c1180cdf8 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/574be8c010074f86a7b8981c1180cdf8 2023-03-31 16:55:15,630 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/3b05da6685e14df5afcdfafef2d786ac to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/3b05da6685e14df5afcdfafef2d786ac 2023-03-31 16:55:15,631 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/f70cf9e46cab42e283d182e9f9a3c1a5 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/f70cf9e46cab42e283d182e9f9a3c1a5 2023-03-31 16:55:15,631 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/c282ae5d607e43baae8fdb9391ac5c10 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/c282ae5d607e43baae8fdb9391ac5c10 2023-03-31 16:55:15,634 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/8a23d40f8d7c46a48e3e5f02e537f0c0, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/e5b3f259af94447a8e2f280b1d9ad17e, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/8f9289fe2d404586b88faedd39c55668, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/b19e36da9bd44e379cc9225b0a44dbdb, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/3d88d8c9451f4f5cad60aff900879f35, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/fa817bcbf06040269857bfc06c6f8a62, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/305c2b13981440039c54d0b08b23188b, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/5302cd4baa30418996ac4ea47c688c8e, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/48ee8e530bf848169a416a1ad8cb40dc, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/2d04dee6d0f94e379b9e1980afdb654a, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/e0026de07c424e348a0b7ebdbd77ebfa, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/db96159877b349c989a32da1e10184f6, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/ca8db8334a614dbbb1b2f33b53104c4d, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/7d142958f1524c54b91d28c5b07e27ad, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/c8594bb4a19448a39ea9b175c21becba, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/5f7fe022293f4aa98ebcb21fff1032dd, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/e4d45c15f55d47e4b719b39f9360daa0, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/0e0b7ffc064d448f8aa3adb0c2f5392a, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/9d517bbd93fc4e7797cfb99815a88baf, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/8fd40085a0e84570b817f748d032fa94, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/7d970e00555e4f21b6013042361665fd, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/d78c305405d8415aaadb5cf7c4b3d302, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/4b5d5801a7f04312870eb1fd5425d988, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/a54162cd97bb4bb09dfe21b2120d9ee5, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/1fc7c61262ba4c35b87a52128ec1e4c9, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/96850842fa774465b0fd2f15d14d0886, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/6ecc9e10289a44be882c6c8f6968e54c, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/47e714c3054542858ae5b4cb9d2b55fe, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/58a795a8bd28468a9f422e74ca6ba07f, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/1555bd52f2b54ddfbbccb2e744fd7669, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/a4d616a3cf1c495bb2334a14d1699c58, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/54706c23a3c04e54b5c67d1dd8ba1c2c, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/33f1d682ece44294a3e4910a2543f38c, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/20113e99ee6f4e898e4438af4bb3e060, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/ea08b127ac704df3a5a5873bf3c55702, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/7eead4bbc30548fba31a7624f1c1eac7, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/34cc6ed295a24d8383b1eedfaddf46b5, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/fdae192f025d4439822ab420380c3884, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/673908c6b2c9419c8b58f5e16d13ef53, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/422bc5ea62bb44c08538569684537aab, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/b71c651c8d224db2be49420bdda3c856, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/96b38dff61494cdda58d1cd885227261, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/4246b31358fe460daea429a11e2bd7a0] to archive 2023-03-31 16:55:15,635 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-31 16:55:15,636 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/8a23d40f8d7c46a48e3e5f02e537f0c0 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/8a23d40f8d7c46a48e3e5f02e537f0c0 2023-03-31 16:55:15,637 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/e5b3f259af94447a8e2f280b1d9ad17e to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/e5b3f259af94447a8e2f280b1d9ad17e 2023-03-31 16:55:15,638 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/8f9289fe2d404586b88faedd39c55668 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/8f9289fe2d404586b88faedd39c55668 2023-03-31 16:55:15,639 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/b19e36da9bd44e379cc9225b0a44dbdb to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/b19e36da9bd44e379cc9225b0a44dbdb 2023-03-31 16:55:15,640 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/3d88d8c9451f4f5cad60aff900879f35 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/3d88d8c9451f4f5cad60aff900879f35 2023-03-31 16:55:15,641 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/fa817bcbf06040269857bfc06c6f8a62 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/fa817bcbf06040269857bfc06c6f8a62 2023-03-31 16:55:15,641 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/305c2b13981440039c54d0b08b23188b to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/305c2b13981440039c54d0b08b23188b 2023-03-31 16:55:15,642 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/5302cd4baa30418996ac4ea47c688c8e to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/5302cd4baa30418996ac4ea47c688c8e 2023-03-31 16:55:15,643 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/48ee8e530bf848169a416a1ad8cb40dc to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/48ee8e530bf848169a416a1ad8cb40dc 2023-03-31 16:55:15,644 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/2d04dee6d0f94e379b9e1980afdb654a to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/2d04dee6d0f94e379b9e1980afdb654a 2023-03-31 16:55:15,645 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/e0026de07c424e348a0b7ebdbd77ebfa to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/e0026de07c424e348a0b7ebdbd77ebfa 2023-03-31 16:55:15,646 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/db96159877b349c989a32da1e10184f6 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/db96159877b349c989a32da1e10184f6 2023-03-31 16:55:15,647 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/ca8db8334a614dbbb1b2f33b53104c4d to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/ca8db8334a614dbbb1b2f33b53104c4d 2023-03-31 16:55:15,647 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/7d142958f1524c54b91d28c5b07e27ad to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/7d142958f1524c54b91d28c5b07e27ad 2023-03-31 16:55:15,648 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/c8594bb4a19448a39ea9b175c21becba to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/c8594bb4a19448a39ea9b175c21becba 2023-03-31 16:55:15,649 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/5f7fe022293f4aa98ebcb21fff1032dd to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/5f7fe022293f4aa98ebcb21fff1032dd 2023-03-31 16:55:15,650 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/e4d45c15f55d47e4b719b39f9360daa0 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/e4d45c15f55d47e4b719b39f9360daa0 2023-03-31 16:55:15,651 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/0e0b7ffc064d448f8aa3adb0c2f5392a to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/0e0b7ffc064d448f8aa3adb0c2f5392a 2023-03-31 16:55:15,651 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/9d517bbd93fc4e7797cfb99815a88baf to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/9d517bbd93fc4e7797cfb99815a88baf 2023-03-31 16:55:15,652 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/8fd40085a0e84570b817f748d032fa94 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/8fd40085a0e84570b817f748d032fa94 2023-03-31 16:55:15,653 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/7d970e00555e4f21b6013042361665fd to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/7d970e00555e4f21b6013042361665fd 2023-03-31 16:55:15,654 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/d78c305405d8415aaadb5cf7c4b3d302 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/d78c305405d8415aaadb5cf7c4b3d302 2023-03-31 16:55:15,655 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/4b5d5801a7f04312870eb1fd5425d988 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/4b5d5801a7f04312870eb1fd5425d988 2023-03-31 16:55:15,655 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/a54162cd97bb4bb09dfe21b2120d9ee5 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/a54162cd97bb4bb09dfe21b2120d9ee5 2023-03-31 16:55:15,656 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/1fc7c61262ba4c35b87a52128ec1e4c9 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/1fc7c61262ba4c35b87a52128ec1e4c9 2023-03-31 16:55:15,657 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/96850842fa774465b0fd2f15d14d0886 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/96850842fa774465b0fd2f15d14d0886 2023-03-31 16:55:15,658 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/6ecc9e10289a44be882c6c8f6968e54c to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/6ecc9e10289a44be882c6c8f6968e54c 2023-03-31 16:55:15,659 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/47e714c3054542858ae5b4cb9d2b55fe to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/47e714c3054542858ae5b4cb9d2b55fe 2023-03-31 16:55:15,660 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/58a795a8bd28468a9f422e74ca6ba07f to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/58a795a8bd28468a9f422e74ca6ba07f 2023-03-31 16:55:15,660 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/1555bd52f2b54ddfbbccb2e744fd7669 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/1555bd52f2b54ddfbbccb2e744fd7669 2023-03-31 16:55:15,661 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/a4d616a3cf1c495bb2334a14d1699c58 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/a4d616a3cf1c495bb2334a14d1699c58 2023-03-31 16:55:15,662 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/54706c23a3c04e54b5c67d1dd8ba1c2c to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/54706c23a3c04e54b5c67d1dd8ba1c2c 2023-03-31 16:55:15,663 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/33f1d682ece44294a3e4910a2543f38c to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/33f1d682ece44294a3e4910a2543f38c 2023-03-31 16:55:15,664 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/20113e99ee6f4e898e4438af4bb3e060 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/20113e99ee6f4e898e4438af4bb3e060 2023-03-31 16:55:15,664 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/ea08b127ac704df3a5a5873bf3c55702 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/ea08b127ac704df3a5a5873bf3c55702 2023-03-31 16:55:15,665 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/7eead4bbc30548fba31a7624f1c1eac7 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/7eead4bbc30548fba31a7624f1c1eac7 2023-03-31 16:55:15,666 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/34cc6ed295a24d8383b1eedfaddf46b5 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/34cc6ed295a24d8383b1eedfaddf46b5 2023-03-31 16:55:15,667 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/fdae192f025d4439822ab420380c3884 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/fdae192f025d4439822ab420380c3884 2023-03-31 16:55:15,667 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/673908c6b2c9419c8b58f5e16d13ef53 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/673908c6b2c9419c8b58f5e16d13ef53 2023-03-31 16:55:15,668 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/422bc5ea62bb44c08538569684537aab to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/422bc5ea62bb44c08538569684537aab 2023-03-31 16:55:15,669 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/b71c651c8d224db2be49420bdda3c856 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/b71c651c8d224db2be49420bdda3c856 2023-03-31 16:55:15,670 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/96b38dff61494cdda58d1cd885227261 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/96b38dff61494cdda58d1cd885227261 2023-03-31 16:55:15,671 DEBUG [StoreCloser-TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/4246b31358fe460daea429a11e2bd7a0 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/4246b31358fe460daea429a11e2bd7a0 2023-03-31 16:55:15,674 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/recovered.edits/1327.seqid, newMaxSeqId=1327, maxSeqId=1 2023-03-31 16:55:15,675 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8. 2023-03-31 16:55:15,676 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 3ab680f7c89c77c9be754a4581df5bc8: 2023-03-31 16:55:15,677 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 3ab680f7c89c77c9be754a4581df5bc8 2023-03-31 16:55:15,677 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=3ab680f7c89c77c9be754a4581df5bc8, regionState=CLOSED 2023-03-31 16:55:15,677 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680281715677"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680281715677"}]},"ts":"1680281715677"} 2023-03-31 16:55:15,679 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=32 2023-03-31 16:55:15,679 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=32, state=SUCCESS; CloseRegionProcedure 3ab680f7c89c77c9be754a4581df5bc8, server=jenkins-hbase4.apache.org,33441,1680281619375 in 1.5490 sec 2023-03-31 16:55:15,680 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=31 2023-03-31 16:55:15,680 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=31, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=3ab680f7c89c77c9be754a4581df5bc8, UNASSIGN in 1.5530 sec 2023-03-31 16:55:15,681 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680281715680"}]},"ts":"1680281715680"} 2023-03-31 16:55:15,681 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-31 16:55:15,683 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-31 16:55:15,684 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=31, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.5620 sec 2023-03-31 16:55:16,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-31 16:55:16,227 INFO [Listener at localhost/44135] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 31 completed 2023-03-31 16:55:16,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-03-31 16:55:16,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure2.ProcedureExecutor(1029): Stored pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-31 16:55:16,230 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-31 16:55:16,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-03-31 16:55:16,231 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=34, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-31 16:55:16,234 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8 2023-03-31 16:55:16,236 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A, FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B, FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C, FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/recovered.edits] 2023-03-31 16:55:16,239 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/bcdba742fa9f4f68b5d9dadaa0b10ef9 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/bcdba742fa9f4f68b5d9dadaa0b10ef9 2023-03-31 16:55:16,240 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/e28e1c5636514d3290f318a6458b78d1 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/A/e28e1c5636514d3290f318a6458b78d1 2023-03-31 16:55:16,242 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/c515673d6b994d9ca27697afac8f2b97 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/c515673d6b994d9ca27697afac8f2b97 2023-03-31 16:55:16,242 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/e7b3c8e310e24e11a751f2d474d9d392 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/B/e7b3c8e310e24e11a751f2d474d9d392 2023-03-31 16:55:16,244 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/2146702411424b0e907d889c6c025a8a to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/2146702411424b0e907d889c6c025a8a 2023-03-31 16:55:16,245 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/5de8de14f82c498fb28160bbf6901c45 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/C/5de8de14f82c498fb28160bbf6901c45 2023-03-31 16:55:16,248 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/recovered.edits/1327.seqid to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8/recovered.edits/1327.seqid 2023-03-31 16:55:16,248 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/3ab680f7c89c77c9be754a4581df5bc8 2023-03-31 16:55:16,248 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-31 16:55:16,250 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=34, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-31 16:55:16,251 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-31 16:55:16,252 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-31 16:55:16,253 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=34, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-31 16:55:16,253 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-31 16:55:16,253 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680281716253"}]},"ts":"9223372036854775807"} 2023-03-31 16:55:16,255 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-31 16:55:16,255 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 3ab680f7c89c77c9be754a4581df5bc8, NAME => 'TestAcidGuarantees,,1680281685194.3ab680f7c89c77c9be754a4581df5bc8.', STARTKEY => '', ENDKEY => ''}] 2023-03-31 16:55:16,255 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-31 16:55:16,255 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680281716255"}]},"ts":"9223372036854775807"} 2023-03-31 16:55:16,256 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-31 16:55:16,257 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=34, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-31 16:55:16,258 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=34, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 29 msec 2023-03-31 16:55:16,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-03-31 16:55:16,332 INFO [Listener at localhost/44135] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 34 completed 2023-03-31 16:55:16,343 INFO [Listener at localhost/44135] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=224 (was 224), OpenFileDescriptor=583 (was 511) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=390 (was 341) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=3538 (was 3730) 2023-03-31 16:55:16,354 INFO [Listener at localhost/44135] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=224, OpenFileDescriptor=583, MaxFileDescriptor=60000, SystemLoadAverage=390, ProcessCount=169, AvailableMemoryMB=3538 2023-03-31 16:55:16,356 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-31 16:55:16,356 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-31 16:55:16,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-31 16:55:16,358 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-31 16:55:16,358 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 35 2023-03-31 16:55:16,358 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-31 16:55:16,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-31 16:55:16,359 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a 2023-03-31 16:55:16,360 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a empty. 2023-03-31 16:55:16,360 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a 2023-03-31 16:55:16,360 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-31 16:55:16,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-31 16:55:16,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-31 16:55:16,770 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-31 16:55:16,771 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 2e08a42276d589261ea162f9850b0f0a, NAME => 'TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp 2023-03-31 16:55:16,776 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-31 16:55:16,776 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 2e08a42276d589261ea162f9850b0f0a, disabling compactions & flushes 2023-03-31 16:55:16,776 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:16,776 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:16,776 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. after waiting 0 ms 2023-03-31 16:55:16,776 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:16,776 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:16,776 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 2e08a42276d589261ea162f9850b0f0a: 2023-03-31 16:55:16,779 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-31 16:55:16,779 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680281716779"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680281716779"}]},"ts":"1680281716779"} 2023-03-31 16:55:16,780 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-31 16:55:16,781 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-31 16:55:16,781 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680281716781"}]},"ts":"1680281716781"} 2023-03-31 16:55:16,782 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-31 16:55:16,785 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2e08a42276d589261ea162f9850b0f0a, ASSIGN}] 2023-03-31 16:55:16,786 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2e08a42276d589261ea162f9850b0f0a, ASSIGN 2023-03-31 16:55:16,787 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=2e08a42276d589261ea162f9850b0f0a, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,33441,1680281619375; forceNewPlan=false, retain=false 2023-03-31 16:55:16,938 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=2e08a42276d589261ea162f9850b0f0a, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:16,938 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680281716938"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680281716938"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680281716938"}]},"ts":"1680281716938"} 2023-03-31 16:55:16,939 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=37, ppid=36, state=RUNNABLE; OpenRegionProcedure 2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375}] 2023-03-31 16:55:16,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-31 16:55:17,093 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:17,094 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 2e08a42276d589261ea162f9850b0f0a, NAME => 'TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a.', STARTKEY => '', ENDKEY => ''} 2023-03-31 16:55:17,094 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 2e08a42276d589261ea162f9850b0f0a 2023-03-31 16:55:17,094 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-31 16:55:17,094 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 2e08a42276d589261ea162f9850b0f0a 2023-03-31 16:55:17,094 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 2e08a42276d589261ea162f9850b0f0a 2023-03-31 16:55:17,095 INFO [StoreOpener-2e08a42276d589261ea162f9850b0f0a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 2e08a42276d589261ea162f9850b0f0a 2023-03-31 16:55:17,096 DEBUG [StoreOpener-2e08a42276d589261ea162f9850b0f0a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A 2023-03-31 16:55:17,096 DEBUG [StoreOpener-2e08a42276d589261ea162f9850b0f0a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A 2023-03-31 16:55:17,096 INFO [StoreOpener-2e08a42276d589261ea162f9850b0f0a-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-31 16:55:17,097 INFO [StoreOpener-2e08a42276d589261ea162f9850b0f0a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2e08a42276d589261ea162f9850b0f0a columnFamilyName A 2023-03-31 16:55:17,097 INFO [StoreOpener-2e08a42276d589261ea162f9850b0f0a-1] regionserver.HStore(310): Store=2e08a42276d589261ea162f9850b0f0a/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-31 16:55:17,097 INFO [StoreOpener-2e08a42276d589261ea162f9850b0f0a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 2e08a42276d589261ea162f9850b0f0a 2023-03-31 16:55:17,098 DEBUG [StoreOpener-2e08a42276d589261ea162f9850b0f0a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B 2023-03-31 16:55:17,098 DEBUG [StoreOpener-2e08a42276d589261ea162f9850b0f0a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B 2023-03-31 16:55:17,098 INFO [StoreOpener-2e08a42276d589261ea162f9850b0f0a-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-31 16:55:17,098 INFO [StoreOpener-2e08a42276d589261ea162f9850b0f0a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2e08a42276d589261ea162f9850b0f0a columnFamilyName B 2023-03-31 16:55:17,099 INFO [StoreOpener-2e08a42276d589261ea162f9850b0f0a-1] regionserver.HStore(310): Store=2e08a42276d589261ea162f9850b0f0a/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-31 16:55:17,099 INFO [StoreOpener-2e08a42276d589261ea162f9850b0f0a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 2e08a42276d589261ea162f9850b0f0a 2023-03-31 16:55:17,100 DEBUG [StoreOpener-2e08a42276d589261ea162f9850b0f0a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C 2023-03-31 16:55:17,100 DEBUG [StoreOpener-2e08a42276d589261ea162f9850b0f0a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C 2023-03-31 16:55:17,100 INFO [StoreOpener-2e08a42276d589261ea162f9850b0f0a-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-31 16:55:17,100 INFO [StoreOpener-2e08a42276d589261ea162f9850b0f0a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2e08a42276d589261ea162f9850b0f0a columnFamilyName C 2023-03-31 16:55:17,100 INFO [StoreOpener-2e08a42276d589261ea162f9850b0f0a-1] regionserver.HStore(310): Store=2e08a42276d589261ea162f9850b0f0a/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-31 16:55:17,100 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:17,101 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a 2023-03-31 16:55:17,101 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a 2023-03-31 16:55:17,102 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-31 16:55:17,103 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 2e08a42276d589261ea162f9850b0f0a 2023-03-31 16:55:17,105 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-31 16:55:17,105 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 2e08a42276d589261ea162f9850b0f0a; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65257136, jitterRate=-0.027592897415161133}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-31 16:55:17,105 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 2e08a42276d589261ea162f9850b0f0a: 2023-03-31 16:55:17,106 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a., pid=37, masterSystemTime=1680281717090 2023-03-31 16:55:17,107 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:17,107 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:17,107 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=2e08a42276d589261ea162f9850b0f0a, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:17,108 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680281717107"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680281717107"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680281717107"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680281717107"}]},"ts":"1680281717107"} 2023-03-31 16:55:17,109 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=37, resume processing ppid=36 2023-03-31 16:55:17,110 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=36, state=SUCCESS; OpenRegionProcedure 2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 in 169 msec 2023-03-31 16:55:17,110 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=36, resume processing ppid=35 2023-03-31 16:55:17,111 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=36, ppid=35, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=2e08a42276d589261ea162f9850b0f0a, ASSIGN in 325 msec 2023-03-31 16:55:17,111 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-31 16:55:17,111 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680281717111"}]},"ts":"1680281717111"} 2023-03-31 16:55:17,112 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-31 16:55:17,115 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-31 16:55:17,116 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 759 msec 2023-03-31 16:55:17,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-31 16:55:17,461 INFO [Listener at localhost/44135] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 35 completed 2023-03-31 16:55:17,462 DEBUG [Listener at localhost/44135] zookeeper.ReadOnlyZKClient(139): Connect 0x4473f45c to 127.0.0.1:51916 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 16:55:17,467 DEBUG [Listener at localhost/44135] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@fd3d54e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 16:55:17,468 DEBUG [Listener at localhost/44135] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 16:55:17,470 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53604, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 16:55:17,471 DEBUG [Listener at localhost/44135] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-31 16:55:17,472 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38594, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-31 16:55:17,473 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-31 16:55:17,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-31 16:55:17,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure2.ProcedureExecutor(1029): Stored pid=38, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-03-31 16:55:17,883 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-31 16:55:17,884 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-31 16:55:17,884 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-31 16:55:17,886 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=39, ppid=38, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-03-31 16:55:17,888 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2e08a42276d589261ea162f9850b0f0a, REOPEN/MOVE}] 2023-03-31 16:55:17,889 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2e08a42276d589261ea162f9850b0f0a, REOPEN/MOVE 2023-03-31 16:55:17,889 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=2e08a42276d589261ea162f9850b0f0a, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:17,889 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680281717889"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680281717889"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680281717889"}]},"ts":"1680281717889"} 2023-03-31 16:55:17,890 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=41, ppid=40, state=RUNNABLE; CloseRegionProcedure 2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375}] 2023-03-31 16:55:18,042 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 2e08a42276d589261ea162f9850b0f0a 2023-03-31 16:55:18,043 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 2e08a42276d589261ea162f9850b0f0a, disabling compactions & flushes 2023-03-31 16:55:18,043 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:18,043 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:18,043 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. after waiting 0 ms 2023-03-31 16:55:18,043 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:18,046 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-03-31 16:55:18,046 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:18,046 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 2e08a42276d589261ea162f9850b0f0a: 2023-03-31 16:55:18,046 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 2e08a42276d589261ea162f9850b0f0a to self. 2023-03-31 16:55:18,047 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 2e08a42276d589261ea162f9850b0f0a 2023-03-31 16:55:18,048 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=2e08a42276d589261ea162f9850b0f0a, regionState=CLOSED 2023-03-31 16:55:18,048 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680281718048"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680281718048"}]},"ts":"1680281718048"} 2023-03-31 16:55:18,050 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=41, resume processing ppid=40 2023-03-31 16:55:18,050 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=41, ppid=40, state=SUCCESS; CloseRegionProcedure 2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 in 159 msec 2023-03-31 16:55:18,050 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=2e08a42276d589261ea162f9850b0f0a, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,33441,1680281619375; forceNewPlan=false, retain=true 2023-03-31 16:55:18,201 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=2e08a42276d589261ea162f9850b0f0a, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:18,201 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680281718201"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680281718201"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680281718201"}]},"ts":"1680281718201"} 2023-03-31 16:55:18,202 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=42, ppid=40, state=RUNNABLE; OpenRegionProcedure 2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375}] 2023-03-31 16:55:18,356 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:18,357 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 2e08a42276d589261ea162f9850b0f0a, NAME => 'TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a.', STARTKEY => '', ENDKEY => ''} 2023-03-31 16:55:18,357 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 2e08a42276d589261ea162f9850b0f0a 2023-03-31 16:55:18,357 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-31 16:55:18,358 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 2e08a42276d589261ea162f9850b0f0a 2023-03-31 16:55:18,358 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 2e08a42276d589261ea162f9850b0f0a 2023-03-31 16:55:18,358 INFO [StoreOpener-2e08a42276d589261ea162f9850b0f0a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 2e08a42276d589261ea162f9850b0f0a 2023-03-31 16:55:18,359 DEBUG [StoreOpener-2e08a42276d589261ea162f9850b0f0a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A 2023-03-31 16:55:18,359 DEBUG [StoreOpener-2e08a42276d589261ea162f9850b0f0a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A 2023-03-31 16:55:18,360 INFO [StoreOpener-2e08a42276d589261ea162f9850b0f0a-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-31 16:55:18,360 INFO [StoreOpener-2e08a42276d589261ea162f9850b0f0a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2e08a42276d589261ea162f9850b0f0a columnFamilyName A 2023-03-31 16:55:18,361 INFO [StoreOpener-2e08a42276d589261ea162f9850b0f0a-1] regionserver.HStore(310): Store=2e08a42276d589261ea162f9850b0f0a/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-31 16:55:18,361 INFO [StoreOpener-2e08a42276d589261ea162f9850b0f0a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 2e08a42276d589261ea162f9850b0f0a 2023-03-31 16:55:18,362 DEBUG [StoreOpener-2e08a42276d589261ea162f9850b0f0a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B 2023-03-31 16:55:18,362 DEBUG [StoreOpener-2e08a42276d589261ea162f9850b0f0a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B 2023-03-31 16:55:18,362 INFO [StoreOpener-2e08a42276d589261ea162f9850b0f0a-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-31 16:55:18,362 INFO [StoreOpener-2e08a42276d589261ea162f9850b0f0a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2e08a42276d589261ea162f9850b0f0a columnFamilyName B 2023-03-31 16:55:18,363 INFO [StoreOpener-2e08a42276d589261ea162f9850b0f0a-1] regionserver.HStore(310): Store=2e08a42276d589261ea162f9850b0f0a/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-31 16:55:18,363 INFO [StoreOpener-2e08a42276d589261ea162f9850b0f0a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 2e08a42276d589261ea162f9850b0f0a 2023-03-31 16:55:18,364 DEBUG [StoreOpener-2e08a42276d589261ea162f9850b0f0a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C 2023-03-31 16:55:18,364 DEBUG [StoreOpener-2e08a42276d589261ea162f9850b0f0a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C 2023-03-31 16:55:18,364 INFO [StoreOpener-2e08a42276d589261ea162f9850b0f0a-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-31 16:55:18,364 INFO [StoreOpener-2e08a42276d589261ea162f9850b0f0a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2e08a42276d589261ea162f9850b0f0a columnFamilyName C 2023-03-31 16:55:18,364 INFO [StoreOpener-2e08a42276d589261ea162f9850b0f0a-1] regionserver.HStore(310): Store=2e08a42276d589261ea162f9850b0f0a/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-31 16:55:18,364 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:18,365 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a 2023-03-31 16:55:18,366 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a 2023-03-31 16:55:18,368 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-31 16:55:18,369 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 2e08a42276d589261ea162f9850b0f0a 2023-03-31 16:55:18,369 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 2e08a42276d589261ea162f9850b0f0a; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60062814, jitterRate=-0.10499432682991028}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-31 16:55:18,369 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 2e08a42276d589261ea162f9850b0f0a: 2023-03-31 16:55:18,370 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a., pid=42, masterSystemTime=1680281718354 2023-03-31 16:55:18,371 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:18,371 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:18,371 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=2e08a42276d589261ea162f9850b0f0a, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:18,371 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680281718371"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680281718371"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680281718371"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680281718371"}]},"ts":"1680281718371"} 2023-03-31 16:55:18,373 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=42, resume processing ppid=40 2023-03-31 16:55:18,373 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=40, state=SUCCESS; OpenRegionProcedure 2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 in 170 msec 2023-03-31 16:55:18,374 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=40, resume processing ppid=39 2023-03-31 16:55:18,374 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=40, ppid=39, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=2e08a42276d589261ea162f9850b0f0a, REOPEN/MOVE in 485 msec 2023-03-31 16:55:18,376 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=39, resume processing ppid=38 2023-03-31 16:55:18,376 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=38, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 489 msec 2023-03-31 16:55:18,378 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=38, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 903 msec 2023-03-31 16:55:18,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1227): Checking to see if procedure is done pid=38 2023-03-31 16:55:18,379 DEBUG [Listener at localhost/44135] zookeeper.ReadOnlyZKClient(139): Connect 0x41507d16 to 127.0.0.1:51916 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 16:55:18,383 DEBUG [Listener at localhost/44135] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1a5ab074, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 16:55:18,384 DEBUG [Listener at localhost/44135] zookeeper.ReadOnlyZKClient(139): Connect 0x49b5bd0d to 127.0.0.1:51916 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 16:55:18,387 DEBUG [Listener at localhost/44135] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@bcfc1e0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 16:55:18,387 DEBUG [Listener at localhost/44135] zookeeper.ReadOnlyZKClient(139): Connect 0x5c8d927d to 127.0.0.1:51916 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 16:55:18,392 DEBUG [Listener at localhost/44135] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@38341134, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 16:55:18,392 DEBUG [Listener at localhost/44135] zookeeper.ReadOnlyZKClient(139): Connect 0x419b69bf to 127.0.0.1:51916 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 16:55:18,396 DEBUG [Listener at localhost/44135] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@71b5058b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 16:55:18,396 DEBUG [Listener at localhost/44135] zookeeper.ReadOnlyZKClient(139): Connect 0x2c6cac96 to 127.0.0.1:51916 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 16:55:18,399 DEBUG [Listener at localhost/44135] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3620ee35, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 16:55:18,399 DEBUG [Listener at localhost/44135] zookeeper.ReadOnlyZKClient(139): Connect 0x4fe1db6a to 127.0.0.1:51916 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 16:55:18,403 DEBUG [Listener at localhost/44135] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7e01d5e5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 16:55:18,403 DEBUG [Listener at localhost/44135] zookeeper.ReadOnlyZKClient(139): Connect 0x10377b50 to 127.0.0.1:51916 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 16:55:18,407 DEBUG [Listener at localhost/44135] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@51ccd8b7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 16:55:18,407 DEBUG [Listener at localhost/44135] zookeeper.ReadOnlyZKClient(139): Connect 0x4366fcdd to 127.0.0.1:51916 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 16:55:18,412 DEBUG [Listener at localhost/44135] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@68a0e21a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 16:55:18,413 DEBUG [Listener at localhost/44135] zookeeper.ReadOnlyZKClient(139): Connect 0x6a35a28d to 127.0.0.1:51916 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 16:55:18,416 DEBUG [Listener at localhost/44135] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1ec12b02, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 16:55:18,416 DEBUG [Listener at localhost/44135] zookeeper.ReadOnlyZKClient(139): Connect 0x5a1d4246 to 127.0.0.1:51916 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 16:55:18,419 DEBUG [Listener at localhost/44135] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1d7ba242, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 16:55:18,426 DEBUG [hconnection-0x27373126-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 16:55:18,426 DEBUG [hconnection-0x2c29380a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 16:55:18,427 DEBUG [hconnection-0x448c20f6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 16:55:18,427 DEBUG [hconnection-0x3de4f08f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 16:55:18,426 DEBUG [hconnection-0x1599d37f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 16:55:18,428 DEBUG [hconnection-0x5e304732-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 16:55:18,428 DEBUG [hconnection-0x689d7782-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 16:55:18,428 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53612, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 16:55:18,430 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53666, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 16:55:18,431 DEBUG [hconnection-0x580a49c5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 16:55:18,431 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53628, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 16:55:18,431 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53640, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 16:55:18,431 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:18,433 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53652, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 16:55:18,433 DEBUG [hconnection-0x139980a0-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 16:55:18,434 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53626, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 16:55:18,434 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53648, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 16:55:18,437 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53686, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 16:55:18,437 DEBUG [hconnection-0x509a2cb-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 16:55:18,437 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53680, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 16:55:18,438 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53692, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 16:55:18,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:18,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:18,441 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:18,443 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:18,443 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:18,444 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:18,444 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:18,444 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:18,447 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:18,447 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:18,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:18,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:18,449 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:18,449 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:18,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:18,449 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:18,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:18,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:18,450 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:18,450 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:18,450 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:18,450 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:18,450 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:18,450 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:18,450 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:18,450 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:18,450 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:18,450 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. started... 2023-03-31 16:55:18,451 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2e08a42276d589261ea162f9850b0f0a 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-03-31 16:55:18,456 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e08a42276d589261ea162f9850b0f0a, store=A 2023-03-31 16:55:18,456 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:18,456 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e08a42276d589261ea162f9850b0f0a, store=B 2023-03-31 16:55:18,456 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:18,456 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e08a42276d589261ea162f9850b0f0a, store=C 2023-03-31 16:55:18,456 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:18,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on 2e08a42276d589261ea162f9850b0f0a 2023-03-31 16:55:18,457 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. as already flushing 2023-03-31 16:55:18,473 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1680281718465, max timestamp=1680281718471Num uniques 859; 2023-03-31 16:55:18,473 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-03-31 16:55:18,473 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1680281718465, max timestamp=1680281718471Num uniques 859; 2023-03-31 16:55:18,473 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-03-31 16:55:18,474 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1680281718465, max timestamp=1680281718471Num uniques 859; 2023-03-31 16:55:18,474 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-03-31 16:55:18,484 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023033139dc441e46f04946b79eb9b9613ee48e to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033139dc441e46f04946b79eb9b9613ee48e 2023-03-31 16:55:18,519 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:18,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53626 deadline: 1680281778512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:18,520 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:18,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53692 deadline: 1680281778513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:18,521 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:18,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53640 deadline: 1680281778513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:18,521 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:18,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53680 deadline: 1680281778514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:18,521 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:18,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53652 deadline: 1680281778519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:18,623 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:18,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53626 deadline: 1680281778620, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:18,623 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:18,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53692 deadline: 1680281778621, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:18,623 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:18,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53640 deadline: 1680281778622, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:18,624 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:18,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53680 deadline: 1680281778622, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:18,624 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:18,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53652 deadline: 1680281778622, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:18,827 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:18,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53680 deadline: 1680281778825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:18,827 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:18,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53640 deadline: 1680281778825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:18,827 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:18,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53626 deadline: 1680281778825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:18,827 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:18,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53692 deadline: 1680281778825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:18,828 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:18,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53652 deadline: 1680281778826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:18,889 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=12, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/A/a6620a784fb843c0b5dac9f70fc7b72f 2023-03-31 16:55:19,131 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:19,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53640 deadline: 1680281779128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:19,131 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:19,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53626 deadline: 1680281779128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:19,131 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:19,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53692 deadline: 1680281779128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:19,131 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:19,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53680 deadline: 1680281779129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:19,132 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:19,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53652 deadline: 1680281779129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:19,299 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/B/3398757e201b4b808ebe5458fbdd4b08 2023-03-31 16:55:19,308 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/C/1f02a13835ae4007b20b6fdd7086e4bb 2023-03-31 16:55:19,312 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/A/a6620a784fb843c0b5dac9f70fc7b72f as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/a6620a784fb843c0b5dac9f70fc7b72f 2023-03-31 16:55:19,315 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/a6620a784fb843c0b5dac9f70fc7b72f, entries=100, sequenceid=12, filesize=18.4 K 2023-03-31 16:55:19,316 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/B/3398757e201b4b808ebe5458fbdd4b08 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/3398757e201b4b808ebe5458fbdd4b08 2023-03-31 16:55:19,319 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/3398757e201b4b808ebe5458fbdd4b08, entries=100, sequenceid=12, filesize=9.3 K 2023-03-31 16:55:19,319 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/C/1f02a13835ae4007b20b6fdd7086e4bb as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/1f02a13835ae4007b20b6fdd7086e4bb 2023-03-31 16:55:19,322 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/1f02a13835ae4007b20b6fdd7086e4bb, entries=100, sequenceid=12, filesize=9.3 K 2023-03-31 16:55:19,323 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=154.31 KB/158010 for 2e08a42276d589261ea162f9850b0f0a in 872ms, sequenceid=12, compaction requested=false 2023-03-31 16:55:19,323 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-03-31 16:55:19,323 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2e08a42276d589261ea162f9850b0f0a: 2023-03-31 16:55:19,323 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:19,323 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:19,323 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:19,324 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:19,324 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:19,324 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:19,325 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:19,325 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:19,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:19,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:19,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:19,325 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:19,325 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:19,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:19,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:19,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:19,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:19,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:19,327 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:19,327 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a0b6a66[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:19,327 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:19,327 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:19,329 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:19,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:19,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:19,329 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:19,329 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:19,329 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:19,329 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:19,329 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:19,330 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:19,330 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:19,330 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:19,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:19,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:19,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:19,330 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:19,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:19,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:19,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:19,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:19,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:19,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:19,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:19,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:19,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:19,333 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:19,333 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:19,333 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:19,333 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:19,333 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:19,335 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:19,335 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:19,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:19,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:19,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:19,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:19,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:19,335 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:19,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:19,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:19,336 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:19,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:19,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:19,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:19,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:19,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:19,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:19,337 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:19,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:19,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:19,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:19,339 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:19,339 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:19,339 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:19,339 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:19,339 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:19,340 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:19,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:19,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:19,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:19,340 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:19,339 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:19,340 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:19,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:19,340 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:19,340 DEBUG [Thread-2550] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:19,340 DEBUG [Thread-2550] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:19,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:19,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:19,440 DEBUG [Thread-2550] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:19,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:19,442 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:19,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:19,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:19,444 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:19,444 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:19,444 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:19,444 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:19,444 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:19,444 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:19,445 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:19,445 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:19,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:19,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:19,446 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:19,446 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:19,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:19,446 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:19,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:19,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:19,447 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:19,447 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:19,447 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:19,447 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:19,447 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:19,447 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:19,447 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:19,447 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:19,447 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:19,447 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. started... 2023-03-31 16:55:19,447 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2e08a42276d589261ea162f9850b0f0a 3/3 column families, dataSize=154.31 KB heapSize=426 KB 2023-03-31 16:55:19,448 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e08a42276d589261ea162f9850b0f0a, store=A 2023-03-31 16:55:19,448 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:55:19,448 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e08a42276d589261ea162f9850b0f0a, store=B 2023-03-31 16:55:19,448 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:55:19,448 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e08a42276d589261ea162f9850b0f0a, store=C 2023-03-31 16:55:19,448 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:55:19,462 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303315898bd95130848da8d496a432bdd19c8 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303315898bd95130848da8d496a432bdd19c8 2023-03-31 16:55:19,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on 2e08a42276d589261ea162f9850b0f0a 2023-03-31 16:55:19,637 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. as already flushing 2023-03-31 16:55:19,642 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:19,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53692 deadline: 1680281779638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:19,642 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:19,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53680 deadline: 1680281779639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:19,642 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:19,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53626 deadline: 1680281779639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:19,644 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:19,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53652 deadline: 1680281779639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:19,645 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:19,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53640 deadline: 1680281779640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:19,745 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:19,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53692 deadline: 1680281779743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:19,747 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:19,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53680 deadline: 1680281779743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:19,747 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:19,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53626 deadline: 1680281779744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:19,748 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:19,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53652 deadline: 1680281779745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:19,748 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:19,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53640 deadline: 1680281779746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:19,867 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=48, memsize=51.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/A/ef864b2a48ed400f9032185238ea5797 2023-03-31 16:55:19,949 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:19,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53692 deadline: 1680281779946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:19,951 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:19,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53626 deadline: 1680281779948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:19,951 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:19,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53640 deadline: 1680281779949, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:19,952 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:19,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53680 deadline: 1680281779949, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:19,952 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:19,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53652 deadline: 1680281779950, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:20,256 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:20,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53692 deadline: 1680281780253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:20,258 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:20,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53680 deadline: 1680281780254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:20,258 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:20,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53626 deadline: 1680281780255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:20,258 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:20,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53640 deadline: 1680281780256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:20,259 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:20,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53652 deadline: 1680281780256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:20,278 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/B/a93a710b824c4cb58db67f7631336f84 2023-03-31 16:55:20,689 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/C/bd01f8213f5f4f6c85bf197afdd0b409 2023-03-31 16:55:20,693 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/A/ef864b2a48ed400f9032185238ea5797 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/ef864b2a48ed400f9032185238ea5797 2023-03-31 16:55:20,696 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/ef864b2a48ed400f9032185238ea5797, entries=150, sequenceid=48, filesize=25.1 K 2023-03-31 16:55:20,697 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/B/a93a710b824c4cb58db67f7631336f84 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/a93a710b824c4cb58db67f7631336f84 2023-03-31 16:55:20,705 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/a93a710b824c4cb58db67f7631336f84, entries=150, sequenceid=48, filesize=11.6 K 2023-03-31 16:55:20,706 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/C/bd01f8213f5f4f6c85bf197afdd0b409 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/bd01f8213f5f4f6c85bf197afdd0b409 2023-03-31 16:55:20,711 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/bd01f8213f5f4f6c85bf197afdd0b409, entries=150, sequenceid=48, filesize=11.6 K 2023-03-31 16:55:20,711 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~425.95 KB/436176, currentSize=33.54 KB/34350 for 2e08a42276d589261ea162f9850b0f0a in 1264ms, sequenceid=48, compaction requested=false 2023-03-31 16:55:20,711 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2e08a42276d589261ea162f9850b0f0a: 2023-03-31 16:55:20,712 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:20,712 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:20,712 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:20,712 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:20,712 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:20,712 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:20,717 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:20,717 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:20,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:20,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:20,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:20,717 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:20,717 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:20,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:20,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:20,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:20,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:20,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:20,719 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:20,719 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3cf82937[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:20,719 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:20,719 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:20,721 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:20,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:20,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:20,721 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:20,721 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:20,721 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:20,721 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:20,721 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:20,722 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:20,722 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:20,722 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:20,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:20,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:20,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:20,722 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:20,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:20,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:20,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:20,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:20,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:20,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:20,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:20,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:20,725 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:20,725 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:20,725 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:20,725 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:20,725 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:20,725 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:20,727 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:20,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:20,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:20,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:20,727 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:20,727 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:20,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:20,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:20,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:20,728 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:20,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:20,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:20,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:20,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:20,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:20,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:20,729 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:20,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:20,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:20,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:20,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:20,731 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:20,731 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:20,731 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:20,731 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:20,731 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:20,732 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:20,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:20,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:20,732 DEBUG [Thread-2550] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:20,732 DEBUG [Thread-2550] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:20,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:20,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:20,732 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:20,732 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:20,732 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:20,732 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:20,732 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:20,732 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:20,732 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:20,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on 2e08a42276d589261ea162f9850b0f0a 2023-03-31 16:55:20,761 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2e08a42276d589261ea162f9850b0f0a 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-31 16:55:20,761 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e08a42276d589261ea162f9850b0f0a, store=A 2023-03-31 16:55:20,761 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:20,761 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e08a42276d589261ea162f9850b0f0a, store=B 2023-03-31 16:55:20,761 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:20,761 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e08a42276d589261ea162f9850b0f0a, store=C 2023-03-31 16:55:20,761 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:20,774 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303313a14f41ddaf54a4e92ec9ed3b3ba7958 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303313a14f41ddaf54a4e92ec9ed3b3ba7958 2023-03-31 16:55:20,778 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=58, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/A/f08fa1a4164348969e5d8d6b70130ddc 2023-03-31 16:55:20,790 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680281720772, max timestamp=1680281720789Num uniques 859; 2023-03-31 16:55:20,790 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680281720772, max timestamp=1680281720789Num uniques 859; 2023-03-31 16:55:20,791 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 16:55:20,791 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 16:55:20,792 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680281720772, max timestamp=1680281720789Num uniques 859; 2023-03-31 16:55:20,792 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 16:55:20,818 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:20,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53652 deadline: 1680281780808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:20,822 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:20,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53640 deadline: 1680281780814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:20,823 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:20,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53680 deadline: 1680281780815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:20,823 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:20,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53692 deadline: 1680281780816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:20,825 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:20,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53626 deadline: 1680281780818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:20,832 DEBUG [Thread-2550] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:20,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:20,834 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:20,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:20,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:20,835 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:20,835 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:20,835 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:20,835 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:20,836 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:20,836 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:20,837 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:20,837 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:20,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:20,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:20,837 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:20,837 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:20,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:20,838 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:20,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:20,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:20,838 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:20,838 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:20,838 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:20,838 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:20,838 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:20,839 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:20,839 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:20,839 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:20,839 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:20,839 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. started... 2023-03-31 16:55:20,839 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. as already flushing 2023-03-31 16:55:20,839 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:20,839 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:20,839 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:20,839 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:20,839 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:20,839 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:20,840 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:20,841 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:20,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:20,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:20,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:20,841 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:20,841 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:20,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:20,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:20,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:20,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:20,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:20,842 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:20,842 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:20,842 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:20,842 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4cbc7028[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:20,845 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:20,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:20,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:20,845 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:20,845 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:20,845 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:20,845 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:20,845 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:20,846 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:20,846 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:20,846 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:20,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:20,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:20,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:20,846 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:20,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:20,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:20,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:20,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:20,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:20,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:20,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:20,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:20,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:20,849 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:20,849 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:20,849 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:20,849 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:20,849 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:20,850 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:20,850 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:20,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:20,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:20,850 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:20,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:20,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:20,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:20,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:20,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:20,851 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:20,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:20,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:20,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:20,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:20,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:20,852 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:20,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:20,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:20,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:20,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:20,854 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:20,854 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:20,854 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:20,855 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:20,855 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:20,854 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:20,855 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:20,855 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:20,855 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:20,855 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:20,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:20,855 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:20,855 DEBUG [Thread-2550] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:20,855 DEBUG [Thread-2550] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:20,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:20,855 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:20,855 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:20,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:20,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:20,924 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:20,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53652 deadline: 1680281780919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:20,924 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:20,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53640 deadline: 1680281780923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:20,925 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:20,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53680 deadline: 1680281780924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:20,925 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:20,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53692 deadline: 1680281780924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:20,931 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:20,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53626 deadline: 1680281780926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:20,955 DEBUG [Thread-2550] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:20,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:20,957 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:20,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:20,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:20,959 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:20,959 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:20,959 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:20,959 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:20,959 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:20,959 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:20,961 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:20,961 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:20,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:20,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:20,961 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:20,961 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:20,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:20,962 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:20,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:20,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:20,962 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:20,962 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:20,962 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:20,962 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:20,962 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:20,962 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:20,962 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:20,962 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:20,962 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:20,962 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. started... 2023-03-31 16:55:20,963 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. as already flushing 2023-03-31 16:55:20,963 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:20,963 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:20,963 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:20,963 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:20,963 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:20,963 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:20,964 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:20,964 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:20,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:20,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:20,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:20,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:20,965 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:20,965 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:20,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:20,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:20,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:20,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:20,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:20,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76725301[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:20,966 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:20,966 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:20,967 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:20,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:20,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:20,967 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:20,967 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:20,967 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:20,967 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:20,968 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:20,969 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:20,969 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:20,969 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:20,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:20,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:20,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:20,969 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:20,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:20,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:20,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:20,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:20,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:20,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:20,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:20,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:20,971 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:20,971 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:20,972 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:20,972 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:20,972 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:20,972 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:20,975 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:20,975 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:20,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:20,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:20,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:20,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:20,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:20,975 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:20,975 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:20,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:20,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:20,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:20,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:20,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:20,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:20,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:20,976 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:20,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:20,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:20,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:20,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:20,978 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:20,978 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:20,978 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:20,978 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:20,978 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:20,978 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:20,978 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:20,979 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:20,979 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:20,979 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:20,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:20,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:20,978 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:20,979 DEBUG [Thread-2550] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:20,979 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:20,979 DEBUG [Thread-2550] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:20,979 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:20,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:20,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:21,079 DEBUG [Thread-2550] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:21,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:21,081 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:21,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:21,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:21,082 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:21,082 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:21,082 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:21,082 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:21,083 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:21,083 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:21,085 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:21,085 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:21,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:21,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:21,085 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:21,085 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:21,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:21,086 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:21,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:21,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:21,087 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:21,087 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:21,087 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:21,088 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:21,088 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:21,088 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:21,088 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:21,088 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:21,088 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:21,088 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. started... 2023-03-31 16:55:21,088 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. as already flushing 2023-03-31 16:55:21,088 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:21,088 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:21,088 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:21,088 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:21,088 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:21,088 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:21,090 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:21,090 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:21,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:21,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:21,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:21,090 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:21,090 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:21,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:21,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:21,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:21,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:21,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:21,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:21,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@376b5ff3[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:21,092 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:21,092 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:21,094 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:21,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:21,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:21,094 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:21,094 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:21,094 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:21,094 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:21,094 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:21,098 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:21,098 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:21,098 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:21,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:21,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:21,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:21,098 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:21,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:21,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:21,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:21,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:21,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:21,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:21,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:21,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:21,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:21,101 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:21,101 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:21,101 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:21,101 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:21,101 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:21,102 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:21,103 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:21,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:21,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:21,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:21,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:21,103 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:21,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:21,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:21,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:21,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:21,103 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:21,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:21,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:21,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:21,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:21,104 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:21,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:21,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:21,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:21,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:21,106 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:21,106 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:21,106 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:21,106 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:21,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:21,106 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:21,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:21,106 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:21,106 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:21,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:21,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:21,106 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:21,107 DEBUG [Thread-2550] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:21,107 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:21,107 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:21,107 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:21,107 DEBUG [Thread-2550] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:21,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:21,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:21,126 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:21,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53652 deadline: 1680281781125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:21,127 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:21,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53640 deadline: 1680281781125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:21,127 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:21,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53692 deadline: 1680281781126, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:21,132 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:21,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53680 deadline: 1680281781126, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:21,138 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:21,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53626 deadline: 1680281781133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:21,194 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/B/83918833daea4429ab630c0475189bc4 2023-03-31 16:55:21,202 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/C/3c31de27ce984fbb8e5d708b806f4f6b 2023-03-31 16:55:21,206 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/A/f08fa1a4164348969e5d8d6b70130ddc as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/f08fa1a4164348969e5d8d6b70130ddc 2023-03-31 16:55:21,207 DEBUG [Thread-2550] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:21,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:21,209 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/f08fa1a4164348969e5d8d6b70130ddc, entries=200, sequenceid=58, filesize=31.9 K 2023-03-31 16:55:21,209 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/B/83918833daea4429ab630c0475189bc4 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/83918833daea4429ab630c0475189bc4 2023-03-31 16:55:21,210 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:21,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:21,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:21,211 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:21,211 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:21,211 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:21,211 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:21,213 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/83918833daea4429ab630c0475189bc4, entries=150, sequenceid=58, filesize=11.6 K 2023-03-31 16:55:21,213 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-31 16:55:21,213 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/C/3c31de27ce984fbb8e5d708b806f4f6b as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/3c31de27ce984fbb8e5d708b806f4f6b 2023-03-31 16:55:21,215 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:21,215 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:21,217 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:21,217 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:21,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:21,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:21,217 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:21,217 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:21,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:21,218 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:21,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:21,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:21,218 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:21,224 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:21,224 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:21,224 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:21,224 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:21,224 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:21,233 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/3c31de27ce984fbb8e5d708b806f4f6b, entries=150, sequenceid=58, filesize=11.6 K 2023-03-31 16:55:21,233 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 2e08a42276d589261ea162f9850b0f0a in 472ms, sequenceid=58, compaction requested=true 2023-03-31 16:55:21,234 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2e08a42276d589261ea162f9850b0f0a: 2023-03-31 16:55:21,234 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:55:21,234 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:55:21,234 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 16:55:21,234 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 16:55:21,234 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-31 16:55:21,235 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 77230 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 16:55:21,235 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1912): 2e08a42276d589261ea162f9850b0f0a/A is initiating minor compaction (all files) 2023-03-31 16:55:21,235 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2e08a42276d589261ea162f9850b0f0a/A in TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:21,235 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/a6620a784fb843c0b5dac9f70fc7b72f, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/ef864b2a48ed400f9032185238ea5797, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/f08fa1a4164348969e5d8d6b70130ddc] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp, totalSize=75.4 K 2023-03-31 16:55:21,235 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33314 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 16:55:21,236 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): 2e08a42276d589261ea162f9850b0f0a/B is initiating minor compaction (all files) 2023-03-31 16:55:21,236 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2e08a42276d589261ea162f9850b0f0a/B in TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:21,236 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/3398757e201b4b808ebe5458fbdd4b08, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/a93a710b824c4cb58db67f7631336f84, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/83918833daea4429ab630c0475189bc4] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp, totalSize=32.5 K 2023-03-31 16:55:21,236 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting a6620a784fb843c0b5dac9f70fc7b72f, keycount=100, bloomtype=ROW, size=18.4 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1680281718449 2023-03-31 16:55:21,237 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 3398757e201b4b808ebe5458fbdd4b08, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1680281718449 2023-03-31 16:55:21,237 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting ef864b2a48ed400f9032185238ea5797, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1680281718511 2023-03-31 16:55:21,237 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting f08fa1a4164348969e5d8d6b70130ddc, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1680281719636 2023-03-31 16:55:21,238 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting a93a710b824c4cb58db67f7631336f84, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1680281718511 2023-03-31 16:55:21,238 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 83918833daea4429ab630c0475189bc4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1680281719636 2023-03-31 16:55:21,242 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:21,242 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:21,242 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:21,242 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. started... 2023-03-31 16:55:21,242 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2e08a42276d589261ea162f9850b0f0a 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-03-31 16:55:21,242 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e08a42276d589261ea162f9850b0f0a, store=A 2023-03-31 16:55:21,242 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:55:21,242 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e08a42276d589261ea162f9850b0f0a, store=B 2023-03-31 16:55:21,242 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:55:21,242 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e08a42276d589261ea162f9850b0f0a, store=C 2023-03-31 16:55:21,243 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:55:21,252 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): 2e08a42276d589261ea162f9850b0f0a#B#compaction#350 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:55:21,269 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2e08a42276d589261ea162f9850b0f0a#A#compaction#351 average throughput is 1.10 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:55:21,285 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023033172a1bf808f674bb8887cffa379a7e5dd to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033172a1bf808f674bb8887cffa379a7e5dd 2023-03-31 16:55:21,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on 2e08a42276d589261ea162f9850b0f0a 2023-03-31 16:55:21,434 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. as already flushing 2023-03-31 16:55:21,443 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:21,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53626 deadline: 1680281781440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:21,447 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:21,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53652 deadline: 1680281781441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:21,449 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:21,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53680 deadline: 1680281781442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:21,450 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:21,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53640 deadline: 1680281781443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:21,450 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:21,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53692 deadline: 1680281781443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:21,551 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:21,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53652 deadline: 1680281781548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:21,556 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:21,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53680 deadline: 1680281781550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:21,556 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:21,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53640 deadline: 1680281781551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:21,556 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:21,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53692 deadline: 1680281781551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:21,682 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/B/e4dc33135f8b4d76b967d8da7b33277d as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/e4dc33135f8b4d76b967d8da7b33277d 2023-03-31 16:55:21,686 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2e08a42276d589261ea162f9850b0f0a/B of 2e08a42276d589261ea162f9850b0f0a into e4dc33135f8b4d76b967d8da7b33277d(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:55:21,686 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2e08a42276d589261ea162f9850b0f0a: 2023-03-31 16:55:21,686 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a., storeName=2e08a42276d589261ea162f9850b0f0a/B, priority=13, startTime=1680281721234; duration=0sec 2023-03-31 16:55:21,686 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:55:21,686 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 16:55:21,687 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33314 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 16:55:21,687 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): 2e08a42276d589261ea162f9850b0f0a/C is initiating minor compaction (all files) 2023-03-31 16:55:21,687 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2e08a42276d589261ea162f9850b0f0a/C in TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:21,687 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/1f02a13835ae4007b20b6fdd7086e4bb, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/bd01f8213f5f4f6c85bf197afdd0b409, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/3c31de27ce984fbb8e5d708b806f4f6b] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp, totalSize=32.5 K 2023-03-31 16:55:21,688 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 1f02a13835ae4007b20b6fdd7086e4bb, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1680281718449 2023-03-31 16:55:21,688 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting bd01f8213f5f4f6c85bf197afdd0b409, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1680281718511 2023-03-31 16:55:21,688 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 3c31de27ce984fbb8e5d708b806f4f6b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1680281719636 2023-03-31 16:55:21,692 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=94, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/A/ff84e982bb2b45989a8687854273af5c 2023-03-31 16:55:21,694 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): 2e08a42276d589261ea162f9850b0f0a#C#compaction#353 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:55:21,705 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/A/7f66cf024b254b04850b29763a69b7fb as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/7f66cf024b254b04850b29763a69b7fb 2023-03-31 16:55:21,705 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=94 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/B/6748de9188374af4b3a1c48fc271be98 2023-03-31 16:55:21,708 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/C/45e8aff6aa3d4ca2a6be13aab773ba71 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/45e8aff6aa3d4ca2a6be13aab773ba71 2023-03-31 16:55:21,712 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2e08a42276d589261ea162f9850b0f0a/A of 2e08a42276d589261ea162f9850b0f0a into 7f66cf024b254b04850b29763a69b7fb(size=25.2 K), total size for store is 25.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:55:21,712 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2e08a42276d589261ea162f9850b0f0a: 2023-03-31 16:55:21,712 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a., storeName=2e08a42276d589261ea162f9850b0f0a/A, priority=13, startTime=1680281721234; duration=0sec 2023-03-31 16:55:21,712 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:55:21,713 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2e08a42276d589261ea162f9850b0f0a/C of 2e08a42276d589261ea162f9850b0f0a into 45e8aff6aa3d4ca2a6be13aab773ba71(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:55:21,713 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2e08a42276d589261ea162f9850b0f0a: 2023-03-31 16:55:21,713 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a., storeName=2e08a42276d589261ea162f9850b0f0a/C, priority=13, startTime=1680281721234; duration=0sec 2023-03-31 16:55:21,713 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:55:21,754 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:21,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53652 deadline: 1680281781753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:21,761 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:21,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53692 deadline: 1680281781757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:21,761 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:21,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53680 deadline: 1680281781758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:21,761 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:21,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53640 deadline: 1680281781758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:21,951 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:21,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53626 deadline: 1680281781948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:22,060 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:22,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53652 deadline: 1680281782055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:22,066 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:22,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53692 deadline: 1680281782062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:22,066 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:22,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53680 deadline: 1680281782062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:22,066 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:22,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53640 deadline: 1680281782064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:22,118 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=94 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/C/03adfdca2a134225883dcc18bf792edd 2023-03-31 16:55:22,121 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/A/ff84e982bb2b45989a8687854273af5c as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/ff84e982bb2b45989a8687854273af5c 2023-03-31 16:55:22,124 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/ff84e982bb2b45989a8687854273af5c, entries=150, sequenceid=94, filesize=25.1 K 2023-03-31 16:55:22,125 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/B/6748de9188374af4b3a1c48fc271be98 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/6748de9188374af4b3a1c48fc271be98 2023-03-31 16:55:22,130 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/6748de9188374af4b3a1c48fc271be98, entries=150, sequenceid=94, filesize=11.6 K 2023-03-31 16:55:22,131 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/C/03adfdca2a134225883dcc18bf792edd as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/03adfdca2a134225883dcc18bf792edd 2023-03-31 16:55:22,134 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/03adfdca2a134225883dcc18bf792edd, entries=150, sequenceid=94, filesize=11.6 K 2023-03-31 16:55:22,134 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 2e08a42276d589261ea162f9850b0f0a in 892ms, sequenceid=94, compaction requested=false 2023-03-31 16:55:22,134 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2e08a42276d589261ea162f9850b0f0a: 2023-03-31 16:55:22,134 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:22,134 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:22,134 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:22,134 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:22,136 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:22,136 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:22,137 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:22,137 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:22,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:22,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:22,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:22,138 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:22,138 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:22,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:22,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:22,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:22,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:22,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:22,140 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:22,140 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4aa779b8[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:22,140 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:22,140 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:22,142 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:22,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:22,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:22,142 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:22,142 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:22,142 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:22,142 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:22,142 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:22,143 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:22,143 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:22,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:22,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:22,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:22,143 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:22,143 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:22,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:22,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:22,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:22,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:22,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:22,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:22,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:22,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:22,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:22,146 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:22,146 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:22,146 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:22,146 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:22,146 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:22,148 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:22,148 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:22,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:22,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:22,148 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:22,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:22,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:22,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:22,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:22,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:22,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:22,150 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:22,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:22,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:22,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:22,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:22,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:22,151 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:22,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:22,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:22,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:22,153 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:22,153 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:22,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:22,153 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:22,153 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:22,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:22,153 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:22,154 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:22,154 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:22,154 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:22,154 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:22,154 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:22,154 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:22,154 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:22,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:22,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:22,154 DEBUG [Thread-2550] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:22,154 DEBUG [Thread-2550] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:22,254 DEBUG [Thread-2550] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:22,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:22,256 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:22,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:22,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:22,257 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:22,257 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:22,258 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:22,258 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:22,258 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:22,258 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:22,259 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:22,259 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:22,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:22,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:22,259 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:22,259 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:22,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:22,260 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:22,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:22,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:22,260 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:22,260 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:22,260 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:22,261 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:22,261 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:22,261 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:22,261 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:22,261 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:22,261 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:22,261 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. started... 2023-03-31 16:55:22,261 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2e08a42276d589261ea162f9850b0f0a 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-31 16:55:22,261 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e08a42276d589261ea162f9850b0f0a, store=A 2023-03-31 16:55:22,261 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:22,262 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e08a42276d589261ea162f9850b0f0a, store=B 2023-03-31 16:55:22,262 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:22,262 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e08a42276d589261ea162f9850b0f0a, store=C 2023-03-31 16:55:22,262 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:22,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on 2e08a42276d589261ea162f9850b0f0a 2023-03-31 16:55:22,568 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. as already flushing 2023-03-31 16:55:22,625 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680281722596, max timestamp=1680281722623Num uniques 859; 2023-03-31 16:55:22,625 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 16:55:22,625 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680281722596, max timestamp=1680281722623Num uniques 859; 2023-03-31 16:55:22,626 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680281722596, max timestamp=1680281722623Num uniques 859; 2023-03-31 16:55:22,626 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 16:55:22,626 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 16:55:22,655 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:22,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53652 deadline: 1680281782649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:22,656 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:22,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53680 deadline: 1680281782654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:22,656 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:22,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53640 deadline: 1680281782654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:22,661 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:22,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53692 deadline: 1680281782655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:22,677 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230331ee96385ef7c04befbce6add257fe7f5e to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331ee96385ef7c04befbce6add257fe7f5e 2023-03-31 16:55:22,766 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:22,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53652 deadline: 1680281782756, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:22,766 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:22,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53680 deadline: 1680281782757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:22,766 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:22,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53640 deadline: 1680281782757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:22,767 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:22,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53692 deadline: 1680281782762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:22,965 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:22,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53626 deadline: 1680281782961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:22,972 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:22,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53640 deadline: 1680281782967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:22,972 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:22,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53652 deadline: 1680281782968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:22,972 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:22,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53680 deadline: 1680281782968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:22,972 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:22,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53692 deadline: 1680281782968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:23,082 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=107, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/A/014d28d712f045e2b3f8956f412b1d7b 2023-03-31 16:55:23,092 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=107 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/B/0e162064eea4434f93abd4ca80c7e0b0 2023-03-31 16:55:23,100 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=107 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/C/52817ba110684de1b130fe2e302ac448 2023-03-31 16:55:23,104 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/A/014d28d712f045e2b3f8956f412b1d7b as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/014d28d712f045e2b3f8956f412b1d7b 2023-03-31 16:55:23,107 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/014d28d712f045e2b3f8956f412b1d7b, entries=150, sequenceid=107, filesize=25.1 K 2023-03-31 16:55:23,108 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/B/0e162064eea4434f93abd4ca80c7e0b0 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/0e162064eea4434f93abd4ca80c7e0b0 2023-03-31 16:55:23,111 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/0e162064eea4434f93abd4ca80c7e0b0, entries=150, sequenceid=107, filesize=11.6 K 2023-03-31 16:55:23,111 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/C/52817ba110684de1b130fe2e302ac448 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/52817ba110684de1b130fe2e302ac448 2023-03-31 16:55:23,114 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/52817ba110684de1b130fe2e302ac448, entries=150, sequenceid=107, filesize=11.6 K 2023-03-31 16:55:23,115 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 2e08a42276d589261ea162f9850b0f0a in 854ms, sequenceid=107, compaction requested=true 2023-03-31 16:55:23,115 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2e08a42276d589261ea162f9850b0f0a: 2023-03-31 16:55:23,115 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:23,115 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:23,115 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:23,115 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:23,115 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:23,115 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:23,117 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:23,117 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:23,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:23,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:23,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:23,117 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:23,117 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:23,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:23,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:23,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:23,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:23,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:23,118 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:23,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65357df4[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:23,119 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:23,119 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:23,121 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:23,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:23,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:23,121 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:23,121 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:23,121 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:23,121 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:23,121 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:23,122 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:23,122 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:23,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:23,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:23,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:23,122 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:23,123 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:23,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:23,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:23,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:23,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:23,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:23,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:23,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:23,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:23,125 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:23,125 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:23,125 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:23,125 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:23,125 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:23,125 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:23,126 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:23,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:23,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:23,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:23,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:23,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:23,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:23,127 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:23,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:23,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:23,127 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:23,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:23,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:23,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:23,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:23,128 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:23,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:23,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:23,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:23,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:23,129 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:23,131 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:23,131 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:23,131 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:23,131 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:23,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:23,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:23,131 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:23,131 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:23,132 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:23,132 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:23,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:23,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:23,132 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:23,132 DEBUG [Thread-2550] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:23,132 DEBUG [Thread-2550] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:23,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:23,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:23,132 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:23,232 DEBUG [Thread-2550] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:23,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:23,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:23,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:23,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:23,235 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:23,235 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:23,235 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:23,235 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:23,236 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:23,236 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:23,242 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:23,242 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:23,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:23,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:23,242 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:23,242 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:23,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:23,243 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:23,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:23,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:23,244 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:23,244 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:23,244 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:23,244 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:23,244 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:23,244 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:23,244 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:23,244 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:23,244 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:23,244 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. started... 2023-03-31 16:55:23,244 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2e08a42276d589261ea162f9850b0f0a 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-03-31 16:55:23,245 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e08a42276d589261ea162f9850b0f0a, store=A 2023-03-31 16:55:23,245 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:55:23,245 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e08a42276d589261ea162f9850b0f0a, store=B 2023-03-31 16:55:23,245 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:55:23,245 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e08a42276d589261ea162f9850b0f0a, store=C 2023-03-31 16:55:23,245 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:55:23,257 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230331060015725ce24b15b742333512e6117a to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331060015725ce24b15b742333512e6117a 2023-03-31 16:55:23,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on 2e08a42276d589261ea162f9850b0f0a 2023-03-31 16:55:23,275 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. as already flushing 2023-03-31 16:55:23,291 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:23,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53680 deadline: 1680281783281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:23,291 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:23,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53652 deadline: 1680281783282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:23,292 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:23,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53692 deadline: 1680281783283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:23,299 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:23,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53640 deadline: 1680281783291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:23,398 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:23,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53680 deadline: 1680281783392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:23,398 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:23,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53652 deadline: 1680281783393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:23,398 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:23,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53692 deadline: 1680281783393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:23,406 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:23,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53640 deadline: 1680281783400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:23,604 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:23,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53680 deadline: 1680281783599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:23,606 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:23,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53692 deadline: 1680281783600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:23,606 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:23,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53652 deadline: 1680281783600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:23,612 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:23,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53640 deadline: 1680281783607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:23,663 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=143, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/A/4a436b177a4a47f9a4b515aabc45e480 2023-03-31 16:55:23,673 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=143 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/B/cc40b230b4c44c57a759febd44a67cde 2023-03-31 16:55:23,682 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=143 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/C/810230a31af8452dbbf98d27ce8beab1 2023-03-31 16:55:23,685 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/A/4a436b177a4a47f9a4b515aabc45e480 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/4a436b177a4a47f9a4b515aabc45e480 2023-03-31 16:55:23,689 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/4a436b177a4a47f9a4b515aabc45e480, entries=150, sequenceid=143, filesize=25.3 K 2023-03-31 16:55:23,689 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/B/cc40b230b4c44c57a759febd44a67cde as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/cc40b230b4c44c57a759febd44a67cde 2023-03-31 16:55:23,693 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/cc40b230b4c44c57a759febd44a67cde, entries=150, sequenceid=143, filesize=11.8 K 2023-03-31 16:55:23,699 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/C/810230a31af8452dbbf98d27ce8beab1 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/810230a31af8452dbbf98d27ce8beab1 2023-03-31 16:55:23,702 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/810230a31af8452dbbf98d27ce8beab1, entries=150, sequenceid=143, filesize=11.8 K 2023-03-31 16:55:23,703 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 2e08a42276d589261ea162f9850b0f0a in 459ms, sequenceid=143, compaction requested=true 2023-03-31 16:55:23,703 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2e08a42276d589261ea162f9850b0f0a: 2023-03-31 16:55:23,703 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:23,703 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:23,703 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:23,703 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:23,703 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:23,703 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:23,706 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:23,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:23,706 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:23,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:23,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:23,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:23,707 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:23,707 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:23,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:23,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:23,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:23,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:23,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:23,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13bd008[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:23,709 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:23,709 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:23,710 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:23,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:23,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:23,710 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:23,710 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:23,710 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:23,712 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:23,712 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:23,712 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:23,712 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:23,712 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:23,712 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:23,712 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:23,712 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:23,712 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:23,712 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:23,712 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:23,720 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:23,720 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:23,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:23,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:23,720 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:23,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:23,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:23,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:23,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:23,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:23,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:23,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:23,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:23,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:23,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:23,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:23,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:23,723 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:23,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:23,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:23,726 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:23,731 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:23,731 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:23,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:23,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:23,732 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:23,732 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:23,732 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:23,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:23,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:23,732 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:23,732 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:23,732 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:23,732 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:23,732 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:23,732 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:23,732 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:23,732 DEBUG [Thread-2550] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:23,732 DEBUG [Thread-2550] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:23,832 DEBUG [Thread-2550] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:23,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:23,834 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:23,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:23,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:23,835 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:23,836 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:23,836 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:23,836 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:23,836 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:23,836 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:23,837 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:23,837 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:23,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:23,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:23,838 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:23,838 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:23,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:23,838 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:23,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:23,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:23,838 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:23,839 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:23,839 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:23,839 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:23,839 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:23,839 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:23,839 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:23,839 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:23,839 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:23,839 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. started... 2023-03-31 16:55:23,839 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2e08a42276d589261ea162f9850b0f0a 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-31 16:55:23,839 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e08a42276d589261ea162f9850b0f0a, store=A 2023-03-31 16:55:23,839 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:23,839 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e08a42276d589261ea162f9850b0f0a, store=B 2023-03-31 16:55:23,839 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:23,839 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e08a42276d589261ea162f9850b0f0a, store=C 2023-03-31 16:55:23,839 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:23,852 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023033112fa8487d63948f2840a8996a0f27ec6 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033112fa8487d63948f2840a8996a0f27ec6 2023-03-31 16:55:23,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on 2e08a42276d589261ea162f9850b0f0a 2023-03-31 16:55:23,911 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. as already flushing 2023-03-31 16:55:23,951 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680281723931, max timestamp=1680281723950Num uniques 859; 2023-03-31 16:55:23,951 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-31 16:55:23,952 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680281723931, max timestamp=1680281723950Num uniques 859; 2023-03-31 16:55:23,952 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680281723931, max timestamp=1680281723950Num uniques 859; 2023-03-31 16:55:23,952 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-31 16:55:23,952 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-31 16:55:24,014 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:24,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53652 deadline: 1680281784005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,014 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:24,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53640 deadline: 1680281784005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,015 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:24,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53692 deadline: 1680281784005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,015 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:24,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53680 deadline: 1680281784006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,126 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:24,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53652 deadline: 1680281784115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,129 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:24,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53640 deadline: 1680281784115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,129 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:24,129 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:24,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53680 deadline: 1680281784116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53692 deadline: 1680281784116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,259 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=153, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/A/0f8be71144434694b159132741a14b7d 2023-03-31 16:55:24,270 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=153 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/B/de2d5603083a45ab87b93250d2806f7a 2023-03-31 16:55:24,278 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=153 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/C/7de3424bc92c45ea9a85199bc283475f 2023-03-31 16:55:24,281 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/A/0f8be71144434694b159132741a14b7d as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/0f8be71144434694b159132741a14b7d 2023-03-31 16:55:24,285 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/0f8be71144434694b159132741a14b7d, entries=150, sequenceid=153, filesize=25.3 K 2023-03-31 16:55:24,285 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/B/de2d5603083a45ab87b93250d2806f7a as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/de2d5603083a45ab87b93250d2806f7a 2023-03-31 16:55:24,289 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/de2d5603083a45ab87b93250d2806f7a, entries=150, sequenceid=153, filesize=11.8 K 2023-03-31 16:55:24,290 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/C/7de3424bc92c45ea9a85199bc283475f as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/7de3424bc92c45ea9a85199bc283475f 2023-03-31 16:55:24,294 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/7de3424bc92c45ea9a85199bc283475f, entries=150, sequenceid=153, filesize=11.8 K 2023-03-31 16:55:24,295 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 2e08a42276d589261ea162f9850b0f0a in 456ms, sequenceid=153, compaction requested=true 2023-03-31 16:55:24,295 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2e08a42276d589261ea162f9850b0f0a: 2023-03-31 16:55:24,295 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:24,295 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:24,295 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:24,295 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,296 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:24,296 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:24,298 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:24,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:24,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:24,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:24,299 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:24,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:24,299 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:24,300 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:24,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:24,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:24,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6cc2831b[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:24,300 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:24,300 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:24,301 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:24,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:24,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:24,302 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:24,302 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:24,302 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:24,302 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,302 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:24,303 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:24,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:24,303 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:24,304 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:24,304 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:24,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:24,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:24,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:24,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:24,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:24,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:24,306 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:24,306 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:24,306 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:24,306 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:24,306 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:24,306 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:24,310 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:24,310 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:24,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:24,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:24,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:24,310 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:24,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:24,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:24,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:24,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:24,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:24,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:24,312 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:24,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:24,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:24,313 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:24,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:24,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,316 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:24,316 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:24,316 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:24,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:24,317 DEBUG [Thread-2550] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:24,317 DEBUG [Thread-2550] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:24,316 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,317 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:24,317 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:24,318 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,318 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:24,318 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:24,318 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:24,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:24,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:24,316 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:24,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:24,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:24,318 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:24,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on 2e08a42276d589261ea162f9850b0f0a 2023-03-31 16:55:24,338 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2e08a42276d589261ea162f9850b0f0a 3/3 column families, dataSize=140.89 KB heapSize=395.53 KB 2023-03-31 16:55:24,338 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e08a42276d589261ea162f9850b0f0a, store=A 2023-03-31 16:55:24,338 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:55:24,338 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e08a42276d589261ea162f9850b0f0a, store=B 2023-03-31 16:55:24,338 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:55:24,338 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e08a42276d589261ea162f9850b0f0a, store=C 2023-03-31 16:55:24,338 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:55:24,355 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:24,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53692 deadline: 1680281784341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,357 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303310ce44b7f778746ae8bbadc0ed2d53991 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303310ce44b7f778746ae8bbadc0ed2d53991 2023-03-31 16:55:24,362 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=189, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/A/5b257cd71994401e8c54fc47ab635770 2023-03-31 16:55:24,365 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:24,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53652 deadline: 1680281784353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,365 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:24,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53640 deadline: 1680281784355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,365 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:24,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53680 deadline: 1680281784356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,417 DEBUG [Thread-2550] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:24,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:24,419 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:24,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:24,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:24,420 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:24,420 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:24,420 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:24,421 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:24,421 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:24,421 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:24,422 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:24,422 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:24,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:24,423 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,423 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:24,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:24,423 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:24,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:24,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:24,423 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:24,424 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:24,424 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:24,424 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:24,424 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:24,424 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:24,428 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:24,428 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:24,428 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:24,428 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. started... 2023-03-31 16:55:24,428 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. as already flushing 2023-03-31 16:55:24,428 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:24,428 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:24,428 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:24,428 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,428 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:24,428 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:24,430 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,430 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:24,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:24,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:24,430 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:24,430 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:24,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:24,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:24,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:24,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:24,432 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:24,432 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1dd95d82[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:24,432 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:24,432 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:24,433 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:24,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:24,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:24,433 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:24,433 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:24,433 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:24,433 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,433 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:24,435 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,436 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:24,436 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:24,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:24,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:24,436 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:24,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:24,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:24,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:24,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:24,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:24,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:24,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:24,438 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:24,438 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:24,438 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:24,438 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:24,438 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:24,440 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:24,440 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:24,440 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:24,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:24,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:24,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:24,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:24,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:24,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:24,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:24,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:24,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:24,441 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:24,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:24,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:24,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:24,442 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:24,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,445 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,445 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:24,445 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:24,445 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:24,445 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:24,445 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:24,445 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:24,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:24,445 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:24,445 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:24,445 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:24,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:24,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:24,445 DEBUG [Thread-2550] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:24,446 DEBUG [Thread-2550] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:24,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:24,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:24,470 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:24,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53692 deadline: 1680281784456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,475 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:24,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53640 deadline: 1680281784466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,475 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:24,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53652 deadline: 1680281784466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,475 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:24,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53680 deadline: 1680281784466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,546 DEBUG [Thread-2550] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:24,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:24,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:24,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:24,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:24,549 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:24,549 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:24,549 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:24,549 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:24,550 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:24,550 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:24,551 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:24,551 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:24,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:24,552 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,552 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:24,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:24,552 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:24,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:24,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:24,552 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:24,553 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:24,553 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:24,553 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:24,553 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:24,554 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:24,554 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:24,554 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:24,554 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:24,554 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. started... 2023-03-31 16:55:24,554 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. as already flushing 2023-03-31 16:55:24,554 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:24,554 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:24,554 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:24,554 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,554 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:24,554 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:24,557 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:24,557 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:24,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:24,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:24,558 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:24,558 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:24,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:24,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:24,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:24,559 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:24,559 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@534d5910[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:24,559 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:24,559 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:24,561 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,561 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:24,561 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:24,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:24,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:24,562 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:24,562 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:24,562 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:24,564 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:24,564 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:24,564 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:24,564 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:24,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:24,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:24,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:24,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:24,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:24,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:24,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:24,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:24,569 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:24,570 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:24,570 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:24,570 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:24,570 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:24,572 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:24,572 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:24,573 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:24,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:24,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:24,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:24,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:24,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:24,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:24,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:24,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:24,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:24,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:24,574 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:24,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:24,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:24,576 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:24,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,579 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,579 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:24,579 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:24,579 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:24,579 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:24,579 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:24,579 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,579 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:24,579 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:24,580 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:24,580 DEBUG [Thread-2550] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:24,580 DEBUG [Thread-2550] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:24,580 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:24,580 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:24,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:24,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:24,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:24,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:24,680 DEBUG [Thread-2550] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:24,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:24,682 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:24,682 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:24,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53692 deadline: 1680281784671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,683 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:24,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53640 deadline: 1680281784676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,683 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:24,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53652 deadline: 1680281784677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,683 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:24,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53680 deadline: 1680281784677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:24,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:24,684 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:24,685 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:24,685 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:24,685 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:24,685 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:24,685 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:24,686 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:24,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:24,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:24,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:24,687 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:24,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:24,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:24,688 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:24,688 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,688 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:24,688 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:24,688 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:24,688 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:24,688 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:24,688 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,688 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:24,688 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:24,688 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:24,688 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:24,688 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. started... 2023-03-31 16:55:24,688 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. as already flushing 2023-03-31 16:55:24,688 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:24,689 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:24,689 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:24,689 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,689 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:24,689 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:24,690 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:24,690 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:24,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:24,690 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:24,690 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:24,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:24,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:24,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:24,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:24,692 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:24,692 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c2978e1[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:24,692 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:24,692 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:24,693 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:24,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:24,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:24,693 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:24,693 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:24,693 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:24,696 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:24,696 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:24,696 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,696 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:24,696 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:24,696 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:24,696 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:24,696 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:24,696 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:24,696 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:24,696 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:24,698 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:24,698 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:24,698 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:24,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:24,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:24,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:24,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:24,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:24,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:24,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:24,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:24,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:24,699 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:24,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:24,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:24,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:24,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:24,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,701 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,703 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:24,703 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:24,703 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:24,704 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:24,703 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:24,704 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:24,704 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:24,704 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:24,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:24,704 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,704 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:24,704 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:24,704 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:24,704 DEBUG [Thread-2550] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:24,704 DEBUG [Thread-2550] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:24,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:24,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:24,778 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=189 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/B/eaa2ce95526a4b3c862ac2ac1f47ce59 2023-03-31 16:55:24,791 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=189 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/C/80ca5efe366e43c3824b35b85432b77e 2023-03-31 16:55:24,797 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/A/5b257cd71994401e8c54fc47ab635770 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/5b257cd71994401e8c54fc47ab635770 2023-03-31 16:55:24,802 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/5b257cd71994401e8c54fc47ab635770, entries=200, sequenceid=189, filesize=32.1 K 2023-03-31 16:55:24,803 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/B/eaa2ce95526a4b3c862ac2ac1f47ce59 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/eaa2ce95526a4b3c862ac2ac1f47ce59 2023-03-31 16:55:24,804 DEBUG [Thread-2550] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:24,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:24,807 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:24,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:24,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:24,808 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:24,808 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:24,808 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/eaa2ce95526a4b3c862ac2ac1f47ce59, entries=150, sequenceid=189, filesize=11.8 K 2023-03-31 16:55:24,808 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:24,808 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:24,809 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:24,809 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:24,809 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/C/80ca5efe366e43c3824b35b85432b77e as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/80ca5efe366e43c3824b35b85432b77e 2023-03-31 16:55:24,810 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:24,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:24,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:24,810 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:24,811 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,811 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:24,811 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:24,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:24,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:24,812 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:24,812 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:24,812 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:24,812 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:24,812 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:24,812 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:24,813 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:24,813 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:24,813 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/80ca5efe366e43c3824b35b85432b77e, entries=150, sequenceid=189, filesize=11.8 K 2023-03-31 16:55:24,813 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:24,813 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. started... 2023-03-31 16:55:24,813 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. as already flushing 2023-03-31 16:55:24,814 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:24,814 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:24,814 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:24,814 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,814 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~395.48 KB/404976, currentSize=40.25 KB/41220 for 2e08a42276d589261ea162f9850b0f0a in 476ms, sequenceid=189, compaction requested=true 2023-03-31 16:55:24,814 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2e08a42276d589261ea162f9850b0f0a: 2023-03-31 16:55:24,814 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:55:24,814 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-31 16:55:24,814 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-31 16:55:24,814 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:55:24,814 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:55:24,816 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 162019 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-31 16:55:24,816 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1912): 2e08a42276d589261ea162f9850b0f0a/A is initiating minor compaction (all files) 2023-03-31 16:55:24,816 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2e08a42276d589261ea162f9850b0f0a/A in TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:24,816 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/7f66cf024b254b04850b29763a69b7fb, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/ff84e982bb2b45989a8687854273af5c, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/014d28d712f045e2b3f8956f412b1d7b, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/4a436b177a4a47f9a4b515aabc45e480, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/0f8be71144434694b159132741a14b7d, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/5b257cd71994401e8c54fc47ab635770] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp, totalSize=158.2 K 2023-03-31 16:55:24,817 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 7f66cf024b254b04850b29763a69b7fb, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1680281719636 2023-03-31 16:55:24,817 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting ff84e982bb2b45989a8687854273af5c, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=94, earliestPutTs=1680281720814 2023-03-31 16:55:24,817 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 014d28d712f045e2b3f8956f412b1d7b, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=107, earliestPutTs=1680281721435 2023-03-31 16:55:24,818 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 4a436b177a4a47f9a4b515aabc45e480, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=143, earliestPutTs=1680281722632 2023-03-31 16:55:24,818 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 0f8be71144434694b159132741a14b7d, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=153, earliestPutTs=1680281723280 2023-03-31 16:55:24,818 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 5b257cd71994401e8c54fc47ab635770, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=189, earliestPutTs=1680281723974 2023-03-31 16:55:24,820 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71869 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-31 16:55:24,820 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): 2e08a42276d589261ea162f9850b0f0a/B is initiating minor compaction (all files) 2023-03-31 16:55:24,820 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2e08a42276d589261ea162f9850b0f0a/B in TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:24,820 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/e4dc33135f8b4d76b967d8da7b33277d, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/6748de9188374af4b3a1c48fc271be98, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/0e162064eea4434f93abd4ca80c7e0b0, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/cc40b230b4c44c57a759febd44a67cde, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/de2d5603083a45ab87b93250d2806f7a, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/eaa2ce95526a4b3c862ac2ac1f47ce59] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp, totalSize=70.2 K 2023-03-31 16:55:24,820 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting e4dc33135f8b4d76b967d8da7b33277d, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1680281719636 2023-03-31 16:55:24,821 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 6748de9188374af4b3a1c48fc271be98, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=94, earliestPutTs=1680281720814 2023-03-31 16:55:24,821 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 0e162064eea4434f93abd4ca80c7e0b0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=107, earliestPutTs=1680281721435 2023-03-31 16:55:24,822 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting cc40b230b4c44c57a759febd44a67cde, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=143, earliestPutTs=1680281722632 2023-03-31 16:55:24,822 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting de2d5603083a45ab87b93250d2806f7a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=153, earliestPutTs=1680281723280 2023-03-31 16:55:24,823 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting eaa2ce95526a4b3c862ac2ac1f47ce59, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=189, earliestPutTs=1680281723983 2023-03-31 16:55:24,824 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:24,824 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:24,827 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:24,827 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:24,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:24,827 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:24,827 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:24,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:24,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:24,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:24,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:24,829 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:24,830 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@748be824[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:24,830 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:24,830 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:24,831 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:24,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:24,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:24,831 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:24,831 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,831 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:24,831 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:24,831 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:24,833 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:24,833 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:24,833 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:24,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:24,833 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:24,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:24,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:24,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:24,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:24,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:24,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:24,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:24,836 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:24,837 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:24,837 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:24,837 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:24,837 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:24,838 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:24,838 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:24,838 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:24,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:24,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:24,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:24,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:24,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:24,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:24,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:24,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:24,839 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:24,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:24,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:24,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:24,841 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:24,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:24,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,844 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,844 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:24,844 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:24,844 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,844 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:24,844 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:24,844 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:24,844 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:24,844 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:24,844 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:24,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:24,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:24,844 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:24,845 DEBUG [Thread-2550] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:24,845 DEBUG [Thread-2550] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:24,844 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:24,844 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:24,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:24,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:24,855 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): 2e08a42276d589261ea162f9850b0f0a#B#compaction#369 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:55:24,860 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2e08a42276d589261ea162f9850b0f0a#A#compaction#368 average throughput is 0.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:55:24,945 DEBUG [Thread-2550] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:24,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:24,947 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:24,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:24,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:24,950 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:24,950 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:24,950 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:24,950 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:24,950 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:24,950 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:24,951 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,951 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:24,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:24,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:24,952 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:24,952 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:24,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:24,953 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:24,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:24,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:24,953 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:24,954 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:24,954 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:24,954 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:24,954 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:24,954 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:24,954 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:24,954 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:24,954 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:24,955 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. started... 2023-03-31 16:55:24,955 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2e08a42276d589261ea162f9850b0f0a 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-03-31 16:55:24,955 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e08a42276d589261ea162f9850b0f0a, store=A 2023-03-31 16:55:24,955 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:24,955 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e08a42276d589261ea162f9850b0f0a, store=B 2023-03-31 16:55:24,955 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:24,955 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e08a42276d589261ea162f9850b0f0a, store=C 2023-03-31 16:55:24,955 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:25,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on 2e08a42276d589261ea162f9850b0f0a 2023-03-31 16:55:25,003 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. as already flushing 2023-03-31 16:55:25,055 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680281725041, max timestamp=1680281725054Num uniques 859; 2023-03-31 16:55:25,055 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-31 16:55:25,056 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680281725041, max timestamp=1680281725054Num uniques 859; 2023-03-31 16:55:25,056 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 16:55:25,056 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680281725041, max timestamp=1680281725054Num uniques 859; 2023-03-31 16:55:25,056 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 16:55:25,098 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680281725082, max timestamp=1680281725094Num uniques 1068; 2023-03-31 16:55:25,098 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680281725082, max timestamp=1680281725094Num uniques 1118; 2023-03-31 16:55:25,098 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-03-31 16:55:25,098 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=7688, suffix heap size=144240,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=168 2023-03-31 16:55:25,101 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680281725082, max timestamp=1680281725094Num uniques 1068; 2023-03-31 16:55:25,101 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-03-31 16:55:25,140 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1680281725115, max timestamp=1680281725138Num uniques 1027; 2023-03-31 16:55:25,140 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=12682, suffix heap size=134776,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=277 2023-03-31 16:55:25,141 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1680281725115, max timestamp=1680281725138Num uniques 1027; 2023-03-31 16:55:25,141 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1680281725115, max timestamp=1680281725138Num uniques 1027; 2023-03-31 16:55:25,141 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=12682, suffix heap size=134776,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=277 2023-03-31 16:55:25,141 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=12682, suffix heap size=134776,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=277 2023-03-31 16:55:25,179 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:25,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53692 deadline: 1680281785168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:25,179 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:25,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53640 deadline: 1680281785169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:25,184 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:25,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53652 deadline: 1680281785178, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:25,194 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:25,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53626 deadline: 1680281785179, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:25,194 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:25,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53680 deadline: 1680281785179, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:25,273 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/B/7c152e43c2fc4113ab8c4bcd81cc8f0f as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/7c152e43c2fc4113ab8c4bcd81cc8f0f 2023-03-31 16:55:25,276 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 2e08a42276d589261ea162f9850b0f0a/B of 2e08a42276d589261ea162f9850b0f0a into 7c152e43c2fc4113ab8c4bcd81cc8f0f(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:55:25,277 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2e08a42276d589261ea162f9850b0f0a: 2023-03-31 16:55:25,277 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a., storeName=2e08a42276d589261ea162f9850b0f0a/B, priority=10, startTime=1680281724814; duration=0sec 2023-03-31 16:55:25,277 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:55:25,277 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-31 16:55:25,278 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71869 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-31 16:55:25,279 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): 2e08a42276d589261ea162f9850b0f0a/C is initiating minor compaction (all files) 2023-03-31 16:55:25,279 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2e08a42276d589261ea162f9850b0f0a/C in TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:25,279 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/45e8aff6aa3d4ca2a6be13aab773ba71, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/03adfdca2a134225883dcc18bf792edd, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/52817ba110684de1b130fe2e302ac448, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/810230a31af8452dbbf98d27ce8beab1, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/7de3424bc92c45ea9a85199bc283475f, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/80ca5efe366e43c3824b35b85432b77e] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp, totalSize=70.2 K 2023-03-31 16:55:25,279 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 45e8aff6aa3d4ca2a6be13aab773ba71, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1680281719636 2023-03-31 16:55:25,284 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 03adfdca2a134225883dcc18bf792edd, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=94, earliestPutTs=1680281720814 2023-03-31 16:55:25,284 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 52817ba110684de1b130fe2e302ac448, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=107, earliestPutTs=1680281721435 2023-03-31 16:55:25,285 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 810230a31af8452dbbf98d27ce8beab1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=143, earliestPutTs=1680281722632 2023-03-31 16:55:25,285 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 7de3424bc92c45ea9a85199bc283475f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=153, earliestPutTs=1680281723280 2023-03-31 16:55:25,286 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 80ca5efe366e43c3824b35b85432b77e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=189, earliestPutTs=1680281723983 2023-03-31 16:55:25,287 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:25,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53692 deadline: 1680281785280, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:25,287 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:25,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53640 deadline: 1680281785280, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:25,294 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:25,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53652 deadline: 1680281785286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:25,301 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): 2e08a42276d589261ea162f9850b0f0a#C#compaction#371 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:55:25,302 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:25,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53626 deadline: 1680281785295, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:25,306 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:25,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53680 deadline: 1680281785296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:25,320 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/A/d58a357d639b475f8d5aa7804a1a3910 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/d58a357d639b475f8d5aa7804a1a3910 2023-03-31 16:55:25,323 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/C/1d62fb3174034eb892d2e8d4db7768a8 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/1d62fb3174034eb892d2e8d4db7768a8 2023-03-31 16:55:25,327 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 2e08a42276d589261ea162f9850b0f0a/A of 2e08a42276d589261ea162f9850b0f0a into d58a357d639b475f8d5aa7804a1a3910(size=25.6 K), total size for store is 25.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:55:25,327 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2e08a42276d589261ea162f9850b0f0a: 2023-03-31 16:55:25,327 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a., storeName=2e08a42276d589261ea162f9850b0f0a/A, priority=10, startTime=1680281724814; duration=0sec 2023-03-31 16:55:25,327 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:55:25,329 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 2e08a42276d589261ea162f9850b0f0a/C of 2e08a42276d589261ea162f9850b0f0a into 1d62fb3174034eb892d2e8d4db7768a8(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:55:25,329 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2e08a42276d589261ea162f9850b0f0a: 2023-03-31 16:55:25,329 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a., storeName=2e08a42276d589261ea162f9850b0f0a/C, priority=10, startTime=1680281724814; duration=0sec 2023-03-31 16:55:25,329 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:55:25,374 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023033147da32ecffe74422808fd20597904b30 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033147da32ecffe74422808fd20597904b30 2023-03-31 16:55:25,493 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:25,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53692 deadline: 1680281785488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:25,493 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:25,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53640 deadline: 1680281785489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:25,500 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:25,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53652 deadline: 1680281785495, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:25,510 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:25,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53626 deadline: 1680281785503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:25,512 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:25,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53680 deadline: 1680281785508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:25,789 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=198, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/A/3639fada1ae14f40822eb6ac85aae320 2023-03-31 16:55:25,799 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:25,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53692 deadline: 1680281785795, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:25,799 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:25,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53640 deadline: 1680281785796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:25,804 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:25,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53652 deadline: 1680281785803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:25,816 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:25,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53626 deadline: 1680281785813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:25,816 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:25,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53680 deadline: 1680281785815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,199 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=198 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/B/7c2bc6ac8ad54b18a0e5d3c6323bee7b 2023-03-31 16:55:26,225 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=198 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/C/1ba89f6be7674b9e9211c2a1e0b4639c 2023-03-31 16:55:26,229 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/A/3639fada1ae14f40822eb6ac85aae320 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/3639fada1ae14f40822eb6ac85aae320 2023-03-31 16:55:26,232 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/3639fada1ae14f40822eb6ac85aae320, entries=100, sequenceid=198, filesize=18.5 K 2023-03-31 16:55:26,233 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/B/7c2bc6ac8ad54b18a0e5d3c6323bee7b as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/7c2bc6ac8ad54b18a0e5d3c6323bee7b 2023-03-31 16:55:26,237 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/7c2bc6ac8ad54b18a0e5d3c6323bee7b, entries=100, sequenceid=198, filesize=9.4 K 2023-03-31 16:55:26,238 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/C/1ba89f6be7674b9e9211c2a1e0b4639c as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/1ba89f6be7674b9e9211c2a1e0b4639c 2023-03-31 16:55:26,242 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/1ba89f6be7674b9e9211c2a1e0b4639c, entries=100, sequenceid=198, filesize=9.4 K 2023-03-31 16:55:26,242 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 2e08a42276d589261ea162f9850b0f0a in 1287ms, sequenceid=198, compaction requested=false 2023-03-31 16:55:26,243 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2e08a42276d589261ea162f9850b0f0a: 2023-03-31 16:55:26,243 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:26,243 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:26,243 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:26,243 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,243 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:26,243 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:26,245 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,245 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:26,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:26,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:26,245 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:26,245 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:26,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:26,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:26,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:26,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:26,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:26,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23698e7a[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:26,247 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:26,247 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:26,248 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:26,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:26,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:26,248 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:26,248 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:26,248 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:26,248 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,248 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:26,251 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:26,251 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:26,251 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:26,251 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:26,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:26,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:26,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:26,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:26,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:26,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:26,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:26,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:26,253 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:26,253 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:26,253 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:26,254 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:26,254 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:26,255 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:26,255 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:26,255 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:26,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:26,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:26,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:26,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:26,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:26,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:26,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:26,257 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:26,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:26,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:26,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:26,258 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:26,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:26,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:26,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,261 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,261 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:26,261 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:26,261 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:26,261 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:26,261 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:26,261 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:26,261 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,261 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:26,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:26,261 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:26,261 DEBUG [Thread-2550] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:26,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:26,261 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:26,261 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:26,261 DEBUG [Thread-2550] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:26,261 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:26,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:26,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:26,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on 2e08a42276d589261ea162f9850b0f0a 2023-03-31 16:55:26,309 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2e08a42276d589261ea162f9850b0f0a 3/3 column families, dataSize=147.60 KB heapSize=413.77 KB 2023-03-31 16:55:26,310 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e08a42276d589261ea162f9850b0f0a, store=A 2023-03-31 16:55:26,310 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-31 16:55:26,310 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e08a42276d589261ea162f9850b0f0a, store=B 2023-03-31 16:55:26,310 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-31 16:55:26,310 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e08a42276d589261ea162f9850b0f0a, store=C 2023-03-31 16:55:26,311 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-31 16:55:26,319 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:26,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53640 deadline: 1680281786317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,324 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303310baf7e1beb1f4b398ba738a595dbb7bd to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303310baf7e1beb1f4b398ba738a595dbb7bd 2023-03-31 16:55:26,329 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:26,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53692 deadline: 1680281786318, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,329 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:26,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53680 deadline: 1680281786318, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,329 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:26,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53652 deadline: 1680281786319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,329 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:26,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53626 deadline: 1680281786320, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,361 DEBUG [Thread-2550] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:26,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:26,363 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:26,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:26,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:26,364 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:26,364 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:26,364 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:26,364 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:26,365 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:26,365 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:26,370 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:26,370 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:26,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:26,370 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,370 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:26,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:26,371 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:26,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:26,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:26,371 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:26,371 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:26,371 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:26,372 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:26,372 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:26,372 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:26,372 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:26,372 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:26,372 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:26,372 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. started... 2023-03-31 16:55:26,372 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. as already flushing 2023-03-31 16:55:26,372 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:26,372 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:26,372 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:26,372 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,372 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:26,372 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:26,374 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:26,374 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:26,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:26,374 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:26,374 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:26,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:26,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:26,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:26,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:26,375 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:26,375 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ad5d6de[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:26,375 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:26,375 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:26,376 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:26,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:26,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:26,377 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:26,377 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:26,377 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:26,377 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,377 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:26,378 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,378 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:26,378 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:26,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:26,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:26,378 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:26,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:26,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:26,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:26,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:26,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:26,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:26,382 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:26,382 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:26,382 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:26,382 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:26,382 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:26,382 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:26,384 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:26,384 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:26,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:26,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:26,384 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:26,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:26,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:26,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:26,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:26,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:26,384 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:26,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:26,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:26,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:26,385 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:26,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:26,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:26,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,388 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,388 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:26,388 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:26,388 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:26,388 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:26,388 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:26,388 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:26,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:26,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:26,388 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:26,388 DEBUG [Thread-2550] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:26,388 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,388 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:26,388 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:26,388 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:26,388 DEBUG [Thread-2550] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:26,388 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:26,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:26,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:26,427 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:26,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53640 deadline: 1680281786421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,437 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:26,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53692 deadline: 1680281786430, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,437 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:26,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53652 deadline: 1680281786430, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,488 DEBUG [Thread-2550] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:26,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:26,490 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:26,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:26,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:26,491 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:26,492 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:26,492 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:26,492 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:26,492 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:26,492 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:26,493 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:26,493 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:26,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:26,494 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,494 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:26,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:26,494 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:26,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:26,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:26,495 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:26,495 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:26,495 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:26,495 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:26,495 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:26,495 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:26,495 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:26,495 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:26,495 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:26,495 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. started... 2023-03-31 16:55:26,495 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. as already flushing 2023-03-31 16:55:26,495 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:26,496 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:26,496 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:26,496 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,496 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:26,496 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:26,497 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,497 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:26,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:26,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:26,497 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:26,497 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:26,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:26,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:26,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:26,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:26,499 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:26,499 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5dfc4b15[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:26,499 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:26,499 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:26,501 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:26,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:26,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:26,501 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:26,501 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:26,501 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:26,501 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,501 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:26,502 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:26,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:26,502 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:26,502 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:26,503 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:26,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:26,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:26,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:26,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:26,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:26,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:26,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:26,504 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:26,504 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:26,505 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:26,505 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:26,505 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:26,506 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:26,506 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:26,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:26,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:26,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:26,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:26,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:26,506 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:26,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:26,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:26,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:26,507 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:26,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:26,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:26,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:26,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:26,508 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:26,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,510 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,511 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:26,511 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:26,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:26,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:26,511 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:26,511 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:26,511 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,511 DEBUG [Thread-2550] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:26,511 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:26,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:26,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:26,511 DEBUG [Thread-2550] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:26,511 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:26,511 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:26,511 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:26,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:26,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:26,611 DEBUG [Thread-2550] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:26,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:26,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:26,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:26,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:26,616 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:26,616 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:26,616 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:26,616 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:26,617 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:26,617 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:26,619 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,619 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:26,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:26,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:26,620 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,620 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:26,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:26,620 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:26,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:26,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:26,621 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:26,621 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:26,621 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:26,621 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:26,621 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:26,621 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:26,621 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:26,621 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:26,621 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:26,621 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. started... 2023-03-31 16:55:26,622 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. as already flushing 2023-03-31 16:55:26,622 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:26,622 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:26,622 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:26,622 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,622 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:26,622 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:26,624 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:26,624 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:26,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:26,624 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:26,624 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:26,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:26,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:26,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:26,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:26,628 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:26,628 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:26,628 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@769d5246[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:26,628 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:26,630 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:26,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:26,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:26,630 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:26,630 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:26,630 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:26,630 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,630 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:26,631 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:26,631 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:26,631 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:26,632 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:26,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:26,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:26,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:26,632 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:26,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:26,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53640 deadline: 1680281786628, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:26,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:26,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:26,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:26,634 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:26,634 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:26,634 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:26,634 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:26,634 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:26,637 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:26,637 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:26,637 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:26,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:26,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:26,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:26,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:26,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:26,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:26,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:26,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:26,638 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:26,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:26,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:26,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:26,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,640 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:26,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:26,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,641 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:26,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53692 deadline: 1680281786638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,642 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:26,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53652 deadline: 1680281786639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,644 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:26,644 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:26,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:26,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:26,644 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,644 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:26,644 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:26,644 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:26,644 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,644 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:26,644 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:26,644 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:26,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:26,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:26,645 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:26,645 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:26,645 DEBUG [Thread-2550] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:26,645 DEBUG [Thread-2550] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:26,728 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=273, memsize=51.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/A/bb2910439e1f4f66b0ba0a460ff16d83 2023-03-31 16:55:26,745 DEBUG [Thread-2550] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:26,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:26,748 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:26,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:26,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:26,750 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:26,750 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:26,750 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:26,751 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:26,751 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:26,751 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:26,752 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,752 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:26,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:26,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:26,753 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,753 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:26,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:26,753 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:26,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:26,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:26,754 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:26,760 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:26,760 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:26,760 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:26,760 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:26,760 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:26,761 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:26,761 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:26,761 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:26,761 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. started... 2023-03-31 16:55:26,761 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. as already flushing 2023-03-31 16:55:26,761 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:26,761 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:26,762 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:26,762 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,762 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:26,762 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:26,766 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,766 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:26,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:26,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:26,767 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:26,767 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:26,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:26,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:26,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:26,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:26,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:26,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19e818b1[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:26,769 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:26,769 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:26,770 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:26,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:26,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:26,771 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:26,771 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:26,771 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,771 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:26,771 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:26,772 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:26,772 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:26,772 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,772 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:26,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:26,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:26,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:26,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:26,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:26,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:26,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:26,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:26,775 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:26,775 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:26,775 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:26,775 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:26,775 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:26,775 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:26,777 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:26,777 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:26,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:26,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:26,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:26,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:26,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:26,777 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:26,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:26,777 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:26,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:26,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:26,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:26,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:26,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:26,778 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:26,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:26,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,782 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:26,782 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:26,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:26,782 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:26,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:26,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:26,782 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:26,782 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:26,782 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:26,782 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:26,783 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,783 DEBUG [Thread-2550] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:26,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:26,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:26,783 DEBUG [Thread-2550] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:26,783 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:26,783 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:26,783 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:26,883 DEBUG [Thread-2550] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:26,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:26,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:26,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:26,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:26,886 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:26,886 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:26,886 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:26,886 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:26,887 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:26,887 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:26,888 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,888 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:26,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:26,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:26,888 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,888 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:26,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:26,889 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:26,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:26,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:26,889 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:26,889 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:26,889 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:26,889 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:26,889 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:26,889 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:26,890 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:26,890 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:26,890 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:26,890 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. started... 2023-03-31 16:55:26,890 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. as already flushing 2023-03-31 16:55:26,890 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:26,890 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:26,890 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:26,890 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,890 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:26,890 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:26,891 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,891 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:26,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:26,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:26,892 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:26,892 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:26,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:26,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:26,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:26,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:26,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:26,893 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:26,893 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:26,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d3f5e74[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:26,894 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:26,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:26,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:26,894 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:26,894 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:26,894 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:26,895 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,895 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:26,897 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,897 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:26,897 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:26,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:26,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:26,897 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:26,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:26,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:26,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:26,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:26,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:26,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:26,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:26,899 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:26,899 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:26,899 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:26,899 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:26,899 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:26,900 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:26,900 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:26,900 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:26,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:26,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:26,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:26,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:26,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:26,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:26,901 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:26,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:26,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:26,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:26,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:26,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:26,902 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:26,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:26,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,904 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,904 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:26,904 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:26,904 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,904 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:26,904 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:26,904 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:26,904 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:26,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:26,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:26,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:26,905 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:26,904 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:26,904 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:26,904 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:26,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:26,905 DEBUG [Thread-2550] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:26,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:26,905 DEBUG [Thread-2550] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:26,938 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:26,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53640 deadline: 1680281786935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,946 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:26,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53652 deadline: 1680281786944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:26,948 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:26,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53692 deadline: 1680281786945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,005 DEBUG [Thread-2550] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:27,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:27,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:27,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:27,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:27,008 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:27,009 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:27,009 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:27,009 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:27,009 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:27,009 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:27,010 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,010 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:27,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:27,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:27,011 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,011 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:27,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:27,011 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:27,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:27,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:27,011 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:27,012 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:27,012 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:27,012 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:27,012 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:27,012 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:27,012 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:27,012 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:27,012 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:27,012 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. started... 2023-03-31 16:55:27,012 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. as already flushing 2023-03-31 16:55:27,012 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:27,012 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:27,012 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:27,012 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,012 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:27,012 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:27,016 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,016 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:27,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:27,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:27,016 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:27,016 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:27,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:27,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:27,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:27,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:27,017 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:27,018 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12a40c25[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:27,018 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:27,018 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:27,019 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:27,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:27,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:27,019 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:27,019 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:27,019 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,019 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:27,019 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:27,020 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,020 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:27,020 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:27,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:27,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:27,020 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:27,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:27,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:27,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:27,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:27,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:27,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:27,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:27,023 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:27,023 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:27,023 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:27,023 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:27,023 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:27,024 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:27,024 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:27,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:27,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:27,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:27,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:27,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:27,024 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:27,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:27,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:27,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:27,025 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:27,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:27,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:27,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:27,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,026 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:27,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:27,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,029 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,029 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:27,029 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:27,029 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:27,030 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:27,030 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,030 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:27,030 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:27,030 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:27,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:27,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:27,029 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:27,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:27,030 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:27,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:27,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:27,030 DEBUG [Thread-2550] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:27,030 DEBUG [Thread-2550] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:27,130 DEBUG [Thread-2550] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:27,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:27,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:27,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:27,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:27,134 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:27,134 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:27,134 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:27,134 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:27,134 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:27,134 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:27,135 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:27,135 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:27,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:27,136 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,136 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:27,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:27,136 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:27,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:27,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:27,137 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:27,137 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:27,137 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:27,137 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:27,137 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:27,137 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:27,137 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:27,137 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:27,137 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:27,137 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. started... 2023-03-31 16:55:27,137 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. as already flushing 2023-03-31 16:55:27,137 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:27,137 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:27,137 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:27,137 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,138 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:27,138 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:27,138 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=273 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/B/59bb0cd23e4345d8b99eb3fb5b54a608 2023-03-31 16:55:27,139 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,139 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:27,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:27,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:27,139 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:27,139 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:27,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:27,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:27,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:27,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:27,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:27,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e377355[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:27,141 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:27,141 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:27,142 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:27,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:27,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:27,143 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:27,143 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:27,143 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:27,143 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,143 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:27,148 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,148 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:27,148 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:27,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:27,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:27,148 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:27,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:27,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:27,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:27,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:27,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:27,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:27,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:27,152 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:27,152 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:27,152 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:27,152 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:27,152 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:27,154 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:27,154 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:27,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:27,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:27,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:27,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:27,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:27,154 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:27,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:27,154 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:27,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:27,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:27,155 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:27,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:27,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:27,157 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,158 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:27,158 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:27,158 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:27,158 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:27,158 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:27,158 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,158 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:27,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:27,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:27,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:27,158 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:27,158 DEBUG [Thread-2550] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:27,158 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:27,158 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:27,158 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:27,158 DEBUG [Thread-2550] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:27,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:27,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:27,159 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:27,258 DEBUG [Thread-2550] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:27,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:27,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:27,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:27,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:27,262 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:27,262 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:27,262 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:27,262 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:27,262 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:27,262 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:27,263 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:27,263 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:27,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:27,264 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:27,264 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:27,264 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:27,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:27,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:27,265 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:27,265 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:27,265 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:27,265 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:27,265 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:27,265 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:27,265 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:27,265 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:27,265 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:27,266 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. started... 2023-03-31 16:55:27,266 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. as already flushing 2023-03-31 16:55:27,266 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:27,266 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:27,266 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:27,266 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,266 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:27,266 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:27,267 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:27,267 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:27,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:27,267 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:27,267 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:27,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:27,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:27,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:27,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:27,269 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:27,269 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:27,269 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5743cfb2[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:27,269 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:27,270 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:27,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:27,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:27,270 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:27,270 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:27,270 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,270 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:27,270 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:27,273 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:27,273 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:27,273 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,273 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:27,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:27,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:27,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:27,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:27,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:27,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:27,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:27,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:27,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:27,275 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:27,275 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:27,275 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:27,275 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:27,275 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:27,276 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:27,276 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:27,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:27,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:27,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:27,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:27,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:27,276 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:27,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:27,277 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:27,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:27,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:27,278 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:27,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:27,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:27,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:27,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:27,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,280 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,280 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:27,280 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:27,280 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:27,281 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:27,281 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,281 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:27,281 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:27,281 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:27,280 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:27,280 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:27,281 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:27,281 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:27,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:27,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:27,281 DEBUG [Thread-2550] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:27,281 DEBUG [Thread-2550] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:27,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:27,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:27,333 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:27,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53626 deadline: 1680281787330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,340 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:27,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53680 deadline: 1680281787336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,381 DEBUG [Thread-2550] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:27,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:27,384 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:27,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:27,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:27,385 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:27,385 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:27,385 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:27,385 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:27,385 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:27,385 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:27,387 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,387 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:27,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:27,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:27,387 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,387 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:27,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:27,387 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:27,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:27,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:27,388 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:27,388 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:27,388 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:27,388 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:27,388 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:27,388 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:27,388 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:27,388 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:27,388 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:27,388 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. started... 2023-03-31 16:55:27,389 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. as already flushing 2023-03-31 16:55:27,389 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:27,389 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:27,389 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:27,389 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,389 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:27,389 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:27,396 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:27,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:27,396 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:27,398 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:27,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:27,398 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:27,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:27,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:27,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:27,400 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:27,400 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d8f7d0[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:27,400 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:27,400 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:27,401 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,401 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:27,402 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:27,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:27,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:27,402 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:27,402 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:27,402 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:27,404 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,404 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:27,404 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:27,404 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:27,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:27,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:27,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:27,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:27,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:27,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:27,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:27,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:27,407 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:27,407 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:27,408 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:27,408 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:27,408 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:27,408 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:27,409 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:27,409 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:27,409 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:27,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:27,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:27,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:27,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:27,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:27,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:27,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:27,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:27,410 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:27,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:27,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:27,412 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:27,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,415 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:27,415 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:27,415 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:27,415 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:27,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:27,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:27,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:27,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:27,415 DEBUG [Thread-2550] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:27,415 DEBUG [Thread-2550] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:27,415 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:27,416 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:27,416 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,416 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:27,416 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:27,416 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:27,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:27,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:27,422 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:27,444 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:27,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53640 deadline: 1680281787442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,454 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:27,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53652 deadline: 1680281787451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,456 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:27,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53692 deadline: 1680281787453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,515 DEBUG [Thread-2550] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:27,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:27,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:27,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:27,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:27,518 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:27,518 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:27,518 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:27,518 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:27,519 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:27,519 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:27,520 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,520 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:27,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:27,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:27,520 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,520 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:27,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:27,521 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:27,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:27,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:27,521 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:27,521 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:27,521 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:27,522 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:27,522 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:27,522 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:27,522 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:27,522 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:27,522 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:27,522 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. started... 2023-03-31 16:55:27,522 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. as already flushing 2023-03-31 16:55:27,522 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:27,522 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:27,522 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:27,522 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,522 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:27,522 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:27,524 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:27,524 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:27,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:27,524 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:27,524 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:27,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:27,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:27,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:27,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:27,525 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:27,525 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32af16ca[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:27,525 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:27,525 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:27,527 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:27,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:27,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:27,527 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:27,527 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:27,527 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,527 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:27,527 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:27,529 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:27,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:27,529 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:27,529 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:27,529 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:27,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:27,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:27,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:27,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:27,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:27,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:27,531 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:27,531 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:27,531 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:27,531 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:27,531 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:27,531 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:27,535 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:27,535 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:27,535 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:27,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:27,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:27,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:27,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:27,535 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:27,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:27,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:27,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:27,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:27,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:27,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:27,537 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:27,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:27,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:27,539 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:27,539 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:27,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:27,539 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:27,539 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:27,539 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:27,539 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:27,539 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:27,540 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:27,540 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,540 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:27,540 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:27,540 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:27,540 DEBUG [Thread-2550] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:27,540 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:27,540 DEBUG [Thread-2550] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:27,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:27,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:27,551 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=273 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/C/5a611fbd198345499fc3295729f872cd 2023-03-31 16:55:27,554 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/A/bb2910439e1f4f66b0ba0a460ff16d83 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/bb2910439e1f4f66b0ba0a460ff16d83 2023-03-31 16:55:27,558 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/bb2910439e1f4f66b0ba0a460ff16d83, entries=250, sequenceid=273, filesize=39.2 K 2023-03-31 16:55:27,558 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/B/59bb0cd23e4345d8b99eb3fb5b54a608 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/59bb0cd23e4345d8b99eb3fb5b54a608 2023-03-31 16:55:27,562 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/59bb0cd23e4345d8b99eb3fb5b54a608, entries=150, sequenceid=273, filesize=11.9 K 2023-03-31 16:55:27,562 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/C/5a611fbd198345499fc3295729f872cd as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/5a611fbd198345499fc3295729f872cd 2023-03-31 16:55:27,565 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/5a611fbd198345499fc3295729f872cd, entries=150, sequenceid=273, filesize=11.9 K 2023-03-31 16:55:27,566 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~434.34 KB/444768, currentSize=26.84 KB/27480 for 2e08a42276d589261ea162f9850b0f0a in 1257ms, sequenceid=273, compaction requested=true 2023-03-31 16:55:27,566 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2e08a42276d589261ea162f9850b0f0a: 2023-03-31 16:55:27,566 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:55:27,566 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:55:27,566 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 16:55:27,566 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 16:55:27,566 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:55:27,567 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34172 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 16:55:27,567 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85232 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 16:55:27,567 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): 2e08a42276d589261ea162f9850b0f0a/B is initiating minor compaction (all files) 2023-03-31 16:55:27,567 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1912): 2e08a42276d589261ea162f9850b0f0a/A is initiating minor compaction (all files) 2023-03-31 16:55:27,567 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2e08a42276d589261ea162f9850b0f0a/B in TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:27,567 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2e08a42276d589261ea162f9850b0f0a/A in TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:27,567 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/7c152e43c2fc4113ab8c4bcd81cc8f0f, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/7c2bc6ac8ad54b18a0e5d3c6323bee7b, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/59bb0cd23e4345d8b99eb3fb5b54a608] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp, totalSize=33.4 K 2023-03-31 16:55:27,567 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/d58a357d639b475f8d5aa7804a1a3910, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/3639fada1ae14f40822eb6ac85aae320, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/bb2910439e1f4f66b0ba0a460ff16d83] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp, totalSize=83.2 K 2023-03-31 16:55:27,568 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 7c152e43c2fc4113ab8c4bcd81cc8f0f, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=189, earliestPutTs=1680281723983 2023-03-31 16:55:27,568 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting d58a357d639b475f8d5aa7804a1a3910, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=189, earliestPutTs=1680281723983 2023-03-31 16:55:27,568 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 7c2bc6ac8ad54b18a0e5d3c6323bee7b, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=198, earliestPutTs=1680281724353 2023-03-31 16:55:27,568 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 3639fada1ae14f40822eb6ac85aae320, keycount=100, bloomtype=ROW, size=18.5 K, encoding=NONE, compression=NONE, seqNum=198, earliestPutTs=1680281724353 2023-03-31 16:55:27,568 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 59bb0cd23e4345d8b99eb3fb5b54a608, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=273, earliestPutTs=1680281725160 2023-03-31 16:55:27,568 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting bb2910439e1f4f66b0ba0a460ff16d83, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=273, earliestPutTs=1680281725160 2023-03-31 16:55:27,585 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2e08a42276d589261ea162f9850b0f0a#A#compaction#377 average throughput is 2.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:55:27,589 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): 2e08a42276d589261ea162f9850b0f0a#B#compaction#378 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:55:27,640 DEBUG [Thread-2550] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:27,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:27,642 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:27,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:27,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:27,643 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:27,643 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:27,643 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:27,643 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:27,644 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:27,644 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:27,647 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,647 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:27,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:27,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:27,647 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:27,647 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:27,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:27,648 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:27,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:27,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:27,648 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:27,649 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:27,649 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:27,649 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:27,649 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:27,649 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:27,649 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:27,649 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:27,649 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:27,649 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. started... 2023-03-31 16:55:27,649 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2e08a42276d589261ea162f9850b0f0a 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-03-31 16:55:27,649 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e08a42276d589261ea162f9850b0f0a, store=A 2023-03-31 16:55:27,650 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:27,650 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e08a42276d589261ea162f9850b0f0a, store=B 2023-03-31 16:55:27,650 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:27,650 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e08a42276d589261ea162f9850b0f0a, store=C 2023-03-31 16:55:27,650 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:28,007 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/B/22b8364570004027919520746f1e95cb as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/22b8364570004027919520746f1e95cb 2023-03-31 16:55:28,011 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2e08a42276d589261ea162f9850b0f0a/B of 2e08a42276d589261ea162f9850b0f0a into 22b8364570004027919520746f1e95cb(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:55:28,011 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2e08a42276d589261ea162f9850b0f0a: 2023-03-31 16:55:28,011 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a., storeName=2e08a42276d589261ea162f9850b0f0a/B, priority=13, startTime=1680281727566; duration=0sec 2023-03-31 16:55:28,011 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:55:28,011 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 16:55:28,012 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34172 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 16:55:28,012 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): 2e08a42276d589261ea162f9850b0f0a/C is initiating minor compaction (all files) 2023-03-31 16:55:28,013 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2e08a42276d589261ea162f9850b0f0a/C in TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:28,013 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/1d62fb3174034eb892d2e8d4db7768a8, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/1ba89f6be7674b9e9211c2a1e0b4639c, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/5a611fbd198345499fc3295729f872cd] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp, totalSize=33.4 K 2023-03-31 16:55:28,013 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 1d62fb3174034eb892d2e8d4db7768a8, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=189, earliestPutTs=1680281723983 2023-03-31 16:55:28,013 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 1ba89f6be7674b9e9211c2a1e0b4639c, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=198, earliestPutTs=1680281724353 2023-03-31 16:55:28,015 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 5a611fbd198345499fc3295729f872cd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=273, earliestPutTs=1680281725160 2023-03-31 16:55:28,020 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): 2e08a42276d589261ea162f9850b0f0a#C#compaction#380 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:55:28,032 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/C/66504eb74c96489ab3110b8d4381fed3 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/66504eb74c96489ab3110b8d4381fed3 2023-03-31 16:55:28,036 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2e08a42276d589261ea162f9850b0f0a/C of 2e08a42276d589261ea162f9850b0f0a into 66504eb74c96489ab3110b8d4381fed3(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:55:28,036 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2e08a42276d589261ea162f9850b0f0a: 2023-03-31 16:55:28,036 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a., storeName=2e08a42276d589261ea162f9850b0f0a/C, priority=13, startTime=1680281727566; duration=0sec 2023-03-31 16:55:28,036 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:55:28,062 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230331f81243e16ce6403bb3fb39933b7feb23 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331f81243e16ce6403bb3fb39933b7feb23 2023-03-31 16:55:28,442 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/A/7c9ee1cd241e4e96b730d24e834a6810 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/7c9ee1cd241e4e96b730d24e834a6810 2023-03-31 16:55:28,446 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2e08a42276d589261ea162f9850b0f0a/A of 2e08a42276d589261ea162f9850b0f0a into 7c9ee1cd241e4e96b730d24e834a6810(size=25.8 K), total size for store is 25.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:55:28,446 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2e08a42276d589261ea162f9850b0f0a: 2023-03-31 16:55:28,446 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a., storeName=2e08a42276d589261ea162f9850b0f0a/A, priority=13, startTime=1680281727566; duration=0sec 2023-03-31 16:55:28,446 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:55:28,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on 2e08a42276d589261ea162f9850b0f0a 2023-03-31 16:55:28,472 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. as already flushing 2023-03-31 16:55:28,479 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=280, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/A/ebcc5156327c453b9aaca0effda9648c 2023-03-31 16:55:28,503 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680281728494, max timestamp=1680281728501Num uniques 859; 2023-03-31 16:55:28,504 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 16:55:28,505 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680281728494, max timestamp=1680281728501Num uniques 859; 2023-03-31 16:55:28,505 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680281728494, max timestamp=1680281728501Num uniques 859; 2023-03-31 16:55:28,505 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-03-31 16:55:28,505 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-03-31 16:55:28,537 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680281728525, max timestamp=1680281728535Num uniques 1009; 2023-03-31 16:55:28,537 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=7688, suffix heap size=132904,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=168 2023-03-31 16:55:28,538 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680281728525, max timestamp=1680281728535Num uniques 1018; 2023-03-31 16:55:28,538 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-03-31 16:55:28,539 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680281728525, max timestamp=1680281728535Num uniques 1009; 2023-03-31 16:55:28,539 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=7688, suffix heap size=132904,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=168 2023-03-31 16:55:28,580 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680281728541, max timestamp=1680281728578Num uniques 1027; 2023-03-31 16:55:28,580 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-03-31 16:55:28,581 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680281728541, max timestamp=1680281728578Num uniques 1077; 2023-03-31 16:55:28,581 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680281728541, max timestamp=1680281728578Num uniques 1027; 2023-03-31 16:55:28,581 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-03-31 16:55:28,581 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-03-31 16:55:28,613 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680281728600, max timestamp=1680281728611Num uniques 1086; 2023-03-31 16:55:28,613 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-03-31 16:55:28,613 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680281728600, max timestamp=1680281728611Num uniques 1086; 2023-03-31 16:55:28,613 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-03-31 16:55:28,614 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680281728600, max timestamp=1680281728611Num uniques 1086; 2023-03-31 16:55:28,614 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-03-31 16:55:28,652 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1680281728633, max timestamp=1680281728650Num uniques 1045; 2023-03-31 16:55:28,652 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-03-31 16:55:28,653 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1680281728633, max timestamp=1680281728650Num uniques 1045; 2023-03-31 16:55:28,653 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-03-31 16:55:28,654 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1680281728633, max timestamp=1680281728650Num uniques 1045; 2023-03-31 16:55:28,654 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-03-31 16:55:28,689 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11624, totalHeapSize=26656, min timestamp=1680281728676, max timestamp=1680281728687Num uniques 1104; 2023-03-31 16:55:28,690 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50554, new segment data size=11624, suffix heap size=142784,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=254 2023-03-31 16:55:28,690 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1680281728676, max timestamp=1680281728687Num uniques 1104; 2023-03-31 16:55:28,690 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11624, totalHeapSize=26656, min timestamp=1680281728676, max timestamp=1680281728687Num uniques 1104; 2023-03-31 16:55:28,690 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=11630, suffix heap size=142784,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=254 2023-03-31 16:55:28,690 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50554, new segment data size=11624, suffix heap size=142784,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=254 2023-03-31 16:55:28,721 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1680281728710, max timestamp=1680281728719Num uniques 1113; 2023-03-31 16:55:28,721 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=9748, suffix heap size=143720,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=213 2023-03-31 16:55:28,722 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1680281728710, max timestamp=1680281728719Num uniques 1113; 2023-03-31 16:55:28,722 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1680281728710, max timestamp=1680281728719Num uniques 1113; 2023-03-31 16:55:28,722 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=9748, suffix heap size=143720,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=213 2023-03-31 16:55:28,722 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=9748, suffix heap size=143720,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=213 2023-03-31 16:55:28,750 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:28,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53692 deadline: 1680281788747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:28,751 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1680281728732, max timestamp=1680281728749Num uniques 1072; 2023-03-31 16:55:28,751 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-03-31 16:55:28,751 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1680281728732, max timestamp=1680281728749Num uniques 1072; 2023-03-31 16:55:28,751 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-03-31 16:55:28,752 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1680281728732, max timestamp=1680281728749Num uniques 1072; 2023-03-31 16:55:28,752 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-03-31 16:55:28,807 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1680281728791, max timestamp=1680281728804Num uniques 1081; 2023-03-31 16:55:28,807 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1680281728791, max timestamp=1680281728804Num uniques 1081; 2023-03-31 16:55:28,807 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=12866, suffix heap size=140392,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=281 2023-03-31 16:55:28,807 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=12866, suffix heap size=140392,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=281 2023-03-31 16:55:28,807 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1680281728791, max timestamp=1680281728804Num uniques 1081; 2023-03-31 16:55:28,807 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=12866, suffix heap size=140392,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=281 2023-03-31 16:55:28,856 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1680281728841, max timestamp=1680281728854Num uniques 1140; 2023-03-31 16:55:28,856 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=52210, new segment data size=8700, suffix heap size=146528,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=190 2023-03-31 16:55:28,857 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1680281728841, max timestamp=1680281728854Num uniques 1140; 2023-03-31 16:55:28,857 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1680281728841, max timestamp=1680281728854Num uniques 1140; 2023-03-31 16:55:28,857 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=52210, new segment data size=8700, suffix heap size=146528,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=190 2023-03-31 16:55:28,857 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=52210, new segment data size=8700, suffix heap size=146528,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=190 2023-03-31 16:55:28,894 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=299, cellSize=13694, totalHeapSize=31336, min timestamp=1680281728878, max timestamp=1680281728892Num uniques 1049; 2023-03-31 16:55:28,894 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=299, cellSize=13694, totalHeapSize=31336, min timestamp=1680281728878, max timestamp=1680281728892Num uniques 1049; 2023-03-31 16:55:28,895 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=13694, suffix heap size=137064,new segment heap size=31336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=299 2023-03-31 16:55:28,895 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=299, cellSize=13694, totalHeapSize=31336, min timestamp=1680281728878, max timestamp=1680281728892Num uniques 1049; 2023-03-31 16:55:28,895 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=13694, suffix heap size=137064,new segment heap size=31336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=299 2023-03-31 16:55:28,895 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=13694, suffix heap size=137064,new segment heap size=31336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=299 2023-03-31 16:55:28,897 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=280 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/B/8768938ae0e2465a9ad32cc15c28b647 2023-03-31 16:55:28,960 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:28,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53692 deadline: 1680281788953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:28,962 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:28,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53652 deadline: 1680281788955, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:28,962 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:28,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53640 deadline: 1680281788956, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,064 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:29,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53692 deadline: 1680281789062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,069 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:29,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53652 deadline: 1680281789063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,070 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:29,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53640 deadline: 1680281789063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,267 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:29,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53692 deadline: 1680281789266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,274 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:29,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53652 deadline: 1680281789272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,275 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:29,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53640 deadline: 1680281789272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,306 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=280 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/C/ed4d02d4245e48bf9392e6059a997c7f 2023-03-31 16:55:29,310 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/A/ebcc5156327c453b9aaca0effda9648c as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/ebcc5156327c453b9aaca0effda9648c 2023-03-31 16:55:29,313 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/ebcc5156327c453b9aaca0effda9648c, entries=100, sequenceid=280, filesize=18.6 K 2023-03-31 16:55:29,313 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/B/8768938ae0e2465a9ad32cc15c28b647 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/8768938ae0e2465a9ad32cc15c28b647 2023-03-31 16:55:29,317 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/8768938ae0e2465a9ad32cc15c28b647, entries=100, sequenceid=280, filesize=9.5 K 2023-03-31 16:55:29,317 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/C/ed4d02d4245e48bf9392e6059a997c7f as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/ed4d02d4245e48bf9392e6059a997c7f 2023-03-31 16:55:29,322 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/ed4d02d4245e48bf9392e6059a997c7f, entries=100, sequenceid=280, filesize=9.5 K 2023-03-31 16:55:29,323 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=154.31 KB/158010 for 2e08a42276d589261ea162f9850b0f0a in 1674ms, sequenceid=280, compaction requested=false 2023-03-31 16:55:29,323 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2e08a42276d589261ea162f9850b0f0a: 2023-03-31 16:55:29,323 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:29,323 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:29,323 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:29,323 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,323 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:29,323 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:29,325 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,325 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:29,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:29,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:29,325 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:29,325 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:29,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:29,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:29,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:29,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:29,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:29,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b77611[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:29,326 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:29,327 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:29,328 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:29,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:29,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:29,329 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:29,329 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:29,329 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:29,329 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,329 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:29,330 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,330 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:29,330 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:29,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:29,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:29,330 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:29,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:29,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:29,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:29,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:29,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:29,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:29,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:29,333 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:29,333 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:29,333 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:29,333 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:29,333 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:29,335 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:29,335 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:29,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:29,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:29,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:29,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:29,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:29,335 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:29,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:29,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:29,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:29,336 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:29,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:29,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:29,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:29,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:29,337 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:29,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,339 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,339 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:29,339 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:29,339 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:29,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:29,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:29,339 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:29,339 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:29,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:29,339 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,339 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:29,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:29,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:29,340 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:29,340 DEBUG [Thread-2550] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:29,340 DEBUG [Thread-2550] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:29,340 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:29,340 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:29,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on 2e08a42276d589261ea162f9850b0f0a 2023-03-31 16:55:29,352 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2e08a42276d589261ea162f9850b0f0a 3/3 column families, dataSize=161.02 KB heapSize=452.25 KB 2023-03-31 16:55:29,353 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e08a42276d589261ea162f9850b0f0a, store=A 2023-03-31 16:55:29,353 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-31 16:55:29,353 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e08a42276d589261ea162f9850b0f0a, store=B 2023-03-31 16:55:29,353 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-31 16:55:29,353 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e08a42276d589261ea162f9850b0f0a, store=C 2023-03-31 16:55:29,353 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-31 16:55:29,372 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023033118baebdd303243baaaa18dba03a2697d to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033118baebdd303243baaaa18dba03a2697d 2023-03-31 16:55:29,382 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=493, memsize=53.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/A/a1e50485a85d41bcb57d2662b361bf2a 2023-03-31 16:55:29,397 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:29,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53626 deadline: 1680281789390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,397 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:29,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53680 deadline: 1680281789390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,440 DEBUG [Thread-2550] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:29,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:29,442 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:29,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:29,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:29,443 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:29,443 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:29,443 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:29,443 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:29,443 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:29,443 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:29,445 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:29,445 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:29,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:29,445 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,445 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:29,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:29,447 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:29,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:29,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:29,447 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:29,447 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:29,447 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:29,447 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:29,448 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:29,448 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:29,448 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:29,448 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:29,448 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:29,448 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. started... 2023-03-31 16:55:29,448 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. as already flushing 2023-03-31 16:55:29,448 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:29,448 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:29,448 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:29,448 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,448 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:29,448 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:29,450 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,450 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:29,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:29,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:29,450 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:29,450 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:29,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:29,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:29,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:29,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:29,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:29,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ec0170a[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:29,452 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:29,452 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:29,454 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:29,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:29,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:29,454 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:29,454 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:29,454 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:29,454 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,455 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:29,456 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:29,456 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:29,456 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,456 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:29,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:29,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:29,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:29,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:29,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:29,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:29,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:29,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:29,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:29,461 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:29,461 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:29,461 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:29,461 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:29,461 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:29,463 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:29,463 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:29,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:29,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:29,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:29,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:29,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:29,463 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:29,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:29,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:29,464 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:29,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:29,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:29,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:29,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:29,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,465 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:29,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:29,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,468 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,468 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:29,469 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:29,469 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,469 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:29,469 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:29,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:29,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:29,469 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:29,469 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:29,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:29,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:29,469 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:29,469 DEBUG [Thread-2550] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:29,469 DEBUG [Thread-2550] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:29,469 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:29,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:29,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:29,503 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:29,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53626 deadline: 1680281789498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,503 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:29,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53680 deadline: 1680281789498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,569 DEBUG [Thread-2550] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:29,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:29,571 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:29,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:29,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:29,572 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:29,572 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:29,572 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:29,572 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:29,573 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:29,573 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:29,574 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:29,574 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:29,574 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53692 deadline: 1680281789570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:29,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:29,575 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,575 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:29,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:29,576 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:29,576 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:29,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53652 deadline: 1680281789575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:29,576 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:29,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:29,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53640 deadline: 1680281789576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,576 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:29,577 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:29,577 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:29,578 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:29,578 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:29,578 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:29,578 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:29,578 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:29,578 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:29,578 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. started... 2023-03-31 16:55:29,578 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. as already flushing 2023-03-31 16:55:29,578 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:29,578 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:29,578 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:29,578 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,578 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:29,578 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:29,580 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,580 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:29,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:29,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:29,580 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:29,580 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:29,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:29,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:29,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:29,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:29,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:29,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9610f75[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:29,581 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:29,581 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:29,583 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:29,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:29,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:29,583 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:29,583 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:29,583 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:29,583 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,583 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:29,584 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,584 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:29,584 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:29,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:29,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:29,584 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:29,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:29,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:29,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:29,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:29,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:29,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:29,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:29,586 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:29,587 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:29,587 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:29,587 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:29,587 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:29,589 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:29,589 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:29,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:29,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:29,589 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:29,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:29,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:29,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:29,590 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:29,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:29,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:29,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:29,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:29,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:29,591 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:29,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:29,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:29,596 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,596 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:29,596 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:29,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:29,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:29,596 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:29,596 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:29,596 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:29,596 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:29,596 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:29,596 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,596 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:29,596 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:29,596 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:29,596 DEBUG [Thread-2550] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:29,596 DEBUG [Thread-2550] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:29,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:29,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:29,596 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:29,696 DEBUG [Thread-2550] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:29,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:29,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:29,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:29,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:29,699 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:29,699 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:29,699 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:29,699 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:29,700 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:29,700 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:29,701 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,701 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:29,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:29,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:29,702 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,702 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:29,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:29,702 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:29,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:29,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:29,703 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:29,703 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:29,703 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:29,703 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:29,703 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:29,703 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:29,703 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:29,703 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:29,703 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:29,703 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. started... 2023-03-31 16:55:29,703 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. as already flushing 2023-03-31 16:55:29,703 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:29,703 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:29,703 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:29,703 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,703 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:29,703 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:29,706 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:29,706 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:29,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:29,706 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:29,706 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:29,706 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:29,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:29,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53626 deadline: 1680281789704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,707 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:29,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53680 deadline: 1680281789705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:29,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:29,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:29,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:29,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43b241d5[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:29,708 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:29,708 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:29,709 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:29,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:29,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:29,709 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,709 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:29,709 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:29,709 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:29,709 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:29,711 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,711 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:29,711 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:29,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:29,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:29,711 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:29,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:29,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:29,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:29,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:29,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:29,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:29,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:29,713 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:29,713 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:29,713 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:29,713 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:29,713 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:29,715 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:29,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:29,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:29,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:29,715 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:29,715 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:29,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:29,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:29,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:29,715 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:29,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:29,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:29,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:29,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:29,716 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:29,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:29,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:29,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,721 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,721 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:29,721 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:29,721 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:29,721 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:29,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:29,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:29,721 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:29,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:29,721 DEBUG [Thread-2550] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:29,721 DEBUG [Thread-2550] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:29,721 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:29,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:29,721 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,722 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:29,722 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:29,722 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:29,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:29,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:29,792 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=493 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/B/bc2831e8666242daa0b2190eaa0e6279 2023-03-31 16:55:29,821 DEBUG [Thread-2550] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:29,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:29,824 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:29,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:29,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:29,826 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:29,826 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:29,826 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:29,826 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:29,827 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:29,827 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:29,828 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:29,828 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:29,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:29,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:29,829 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:29,829 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,829 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:29,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:29,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:29,829 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:29,830 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:29,830 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:29,830 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:29,830 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:29,830 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:29,830 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:29,830 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:29,830 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:29,830 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. started... 2023-03-31 16:55:29,830 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. as already flushing 2023-03-31 16:55:29,831 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:29,831 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:29,831 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:29,831 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,831 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:29,831 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:29,832 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,832 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:29,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:29,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:29,833 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:29,833 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:29,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:29,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:29,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:29,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:29,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:29,835 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f09b04c[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:29,835 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:29,835 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:29,836 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:29,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:29,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:29,836 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:29,836 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:29,836 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:29,836 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,836 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:29,841 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:29,841 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:29,841 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,841 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:29,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:29,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:29,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:29,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:29,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:29,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:29,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:29,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:29,843 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:29,843 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:29,843 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:29,843 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:29,843 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:29,843 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:29,845 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:29,845 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:29,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:29,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:29,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:29,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:29,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:29,845 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:29,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:29,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:29,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:29,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:29,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:29,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:29,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:29,847 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:29,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:29,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,848 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,850 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:29,850 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:29,850 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,850 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:29,850 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:29,850 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:29,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:29,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:29,850 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:29,850 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:29,851 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,851 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:29,851 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:29,851 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:29,851 DEBUG [Thread-2550] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:29,851 DEBUG [Thread-2550] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:29,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:29,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:29,951 DEBUG [Thread-2550] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:29,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:29,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:29,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:29,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:29,954 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:29,954 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:29,954 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:29,954 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:29,955 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:29,955 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:29,956 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:29,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:29,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:29,956 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,957 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,957 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:29,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:29,957 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:29,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:29,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:29,957 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:29,958 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:29,958 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:29,958 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:29,958 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:29,958 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:29,958 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:29,958 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:29,958 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. started... 2023-03-31 16:55:29,958 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. as already flushing 2023-03-31 16:55:29,958 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:29,958 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:29,958 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:29,958 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,958 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:29,958 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:29,961 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:29,961 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:29,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:29,961 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:29,961 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:29,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:29,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:29,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:29,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:29,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:29,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@313f90e0[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:29,962 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:29,963 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:29,964 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:29,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:29,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:29,964 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:29,964 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:29,964 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,964 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:29,964 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:29,965 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:29,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:29,965 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:29,965 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:29,966 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:29,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:29,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:29,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:29,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:29,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:29,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:29,968 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:29,968 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:29,968 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:29,968 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:29,968 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:29,968 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:29,970 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:29,970 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:29,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:29,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:29,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:29,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:29,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:29,970 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:29,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:29,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:29,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:29,970 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:29,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:29,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:29,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:29,971 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:29,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:29,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,977 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:29,977 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:29,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:29,977 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:29,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:29,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:29,977 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:29,977 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:29,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:29,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:29,978 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:29,978 DEBUG [Thread-2550] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:29,978 DEBUG [Thread-2550] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:29,978 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:29,978 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:29,978 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:29,978 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:30,011 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:30,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53680 deadline: 1680281790007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:30,012 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:30,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53626 deadline: 1680281790009, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:30,078 DEBUG [Thread-2550] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:30,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:30,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:30,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:30,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:30,081 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:30,081 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:30,081 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:30,081 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:30,081 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:30,081 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:30,083 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:30,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:30,083 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:30,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:30,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:30,083 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:30,083 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:30,083 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:30,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:30,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:30,084 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:30,084 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:30,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53652 deadline: 1680281790078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:30,084 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:30,084 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:30,084 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:30,084 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:30,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53692 deadline: 1680281790078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:30,084 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:30,084 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:30,084 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:30,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53640 deadline: 1680281790081, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:30,085 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:30,085 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:30,085 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:30,085 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. started... 2023-03-31 16:55:30,085 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. as already flushing 2023-03-31 16:55:30,085 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:30,085 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:30,085 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:30,085 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:30,085 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:30,085 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:30,087 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:30,087 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:30,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:30,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:30,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:30,087 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:30,087 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:30,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:30,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:30,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:30,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:30,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:30,088 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:30,088 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30a43b15[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:30,088 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:30,088 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:30,089 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:30,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:30,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:30,089 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:30,089 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:30,089 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:30,089 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:30,089 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:30,091 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:30,091 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:30,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:30,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:30,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:30,091 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:30,092 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:30,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:30,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:30,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:30,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:30,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:30,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:30,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:30,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:30,094 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:30,094 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:30,094 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:30,094 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:30,094 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:30,094 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:30,095 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:30,095 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:30,095 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:30,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:30,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:30,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:30,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:30,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:30,096 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:30,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:30,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:30,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:30,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:30,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:30,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:30,104 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:30,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:30,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:30,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:30,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:30,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:30,107 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:30,107 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:30,107 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:30,107 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:30,107 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:30,107 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:30,107 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:30,107 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:30,107 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:30,107 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:30,107 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:30,107 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:30,107 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:30,107 DEBUG [Thread-2550] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:30,107 DEBUG [Thread-2550] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:30,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:30,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:30,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:30,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:30,202 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=493 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/C/bf4b5703a27849f7abbf04a28df9a49c 2023-03-31 16:55:30,206 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/A/a1e50485a85d41bcb57d2662b361bf2a as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/a1e50485a85d41bcb57d2662b361bf2a 2023-03-31 16:55:30,208 DEBUG [Thread-2550] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:30,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:30,209 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/a1e50485a85d41bcb57d2662b361bf2a, entries=200, sequenceid=493, filesize=32.3 K 2023-03-31 16:55:30,210 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/B/bc2831e8666242daa0b2190eaa0e6279 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/bc2831e8666242daa0b2190eaa0e6279 2023-03-31 16:55:30,210 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:30,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:30,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:30,211 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:30,211 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:30,211 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:30,212 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:30,212 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:30,212 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:30,213 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:30,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:30,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:30,213 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/bc2831e8666242daa0b2190eaa0e6279, entries=150, sequenceid=493, filesize=11.9 K 2023-03-31 16:55:30,213 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:30,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:30,214 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/C/bf4b5703a27849f7abbf04a28df9a49c as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/bf4b5703a27849f7abbf04a28df9a49c 2023-03-31 16:55:30,215 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:30,215 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:30,215 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:30,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:30,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:30,215 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:30,216 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:30,216 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:30,216 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:30,216 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:30,216 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:30,216 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:30,216 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:30,216 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:30,216 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. started... 2023-03-31 16:55:30,216 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. as already flushing 2023-03-31 16:55:30,216 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:30,216 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:30,216 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:30,216 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:30,217 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:30,217 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:30,218 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/bf4b5703a27849f7abbf04a28df9a49c, entries=150, sequenceid=493, filesize=11.9 K 2023-03-31 16:55:30,219 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~161.02 KB/164880, heapSize ~452.20 KB/463056, currentSize=26.84 KB/27480 for 2e08a42276d589261ea162f9850b0f0a in 866ms, sequenceid=493, compaction requested=true 2023-03-31 16:55:30,219 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2e08a42276d589261ea162f9850b0f0a: 2023-03-31 16:55:30,219 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:55:30,219 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 16:55:30,219 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 16:55:30,219 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:55:30,219 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:55:30,219 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:30,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:30,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:30,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:30,219 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:30,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:30,220 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34524 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 16:55:30,220 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): 2e08a42276d589261ea162f9850b0f0a/B is initiating minor compaction (all files) 2023-03-31 16:55:30,220 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2e08a42276d589261ea162f9850b0f0a/B in TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:30,220 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/22b8364570004027919520746f1e95cb, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/8768938ae0e2465a9ad32cc15c28b647, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/bc2831e8666242daa0b2190eaa0e6279] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp, totalSize=33.7 K 2023-03-31 16:55:30,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:30,225 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:30,225 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:30,225 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 78540 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 16:55:30,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:30,225 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1912): 2e08a42276d589261ea162f9850b0f0a/A is initiating minor compaction (all files) 2023-03-31 16:55:30,226 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2e08a42276d589261ea162f9850b0f0a/A in TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:30,226 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/7c9ee1cd241e4e96b730d24e834a6810, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/ebcc5156327c453b9aaca0effda9648c, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/a1e50485a85d41bcb57d2662b361bf2a] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp, totalSize=76.7 K 2023-03-31 16:55:30,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:30,226 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 7c9ee1cd241e4e96b730d24e834a6810, keycount=150, bloomtype=ROW, size=25.8 K, encoding=NONE, compression=NONE, seqNum=273, earliestPutTs=1680281725160 2023-03-31 16:55:30,226 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting ebcc5156327c453b9aaca0effda9648c, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=280, earliestPutTs=1680281726317 2023-03-31 16:55:30,227 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting a1e50485a85d41bcb57d2662b361bf2a, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=493, earliestPutTs=1680281728952 2023-03-31 16:55:30,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:30,229 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:30,229 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61c9683d[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:30,229 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:30,229 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:30,229 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 22b8364570004027919520746f1e95cb, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=273, earliestPutTs=1680281725160 2023-03-31 16:55:30,230 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 8768938ae0e2465a9ad32cc15c28b647, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=280, earliestPutTs=1680281726317 2023-03-31 16:55:30,230 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:30,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:30,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:30,230 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:30,231 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:30,231 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:30,231 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:30,231 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:30,231 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting bc2831e8666242daa0b2190eaa0e6279, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=493, earliestPutTs=1680281728952 2023-03-31 16:55:30,233 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:30,233 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:30,233 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:30,233 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:30,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:30,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:30,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:30,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:30,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:30,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:30,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:30,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:30,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:30,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:30,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:30,239 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:30,239 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:30,239 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:30,239 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:30,239 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:30,239 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:30,241 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:30,241 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:30,241 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:30,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:30,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:30,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:30,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:30,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:30,242 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:30,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:30,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:30,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:30,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:30,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:30,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:30,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:30,244 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:30,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:30,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:30,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:30,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:30,247 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): 2e08a42276d589261ea162f9850b0f0a#B#compaction#387 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:55:30,248 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2e08a42276d589261ea162f9850b0f0a#A#compaction#386 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:55:30,249 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:30,249 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:30,249 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:30,249 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:30,249 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:30,249 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:30,249 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:30,249 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:30,249 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:30,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:30,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:30,249 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:30,249 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:30,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:30,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:30,249 DEBUG [Thread-2550] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:30,249 DEBUG [Thread-2550] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:30,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:30,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:30,269 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/B/0647850e30f64b9489f501cf0d5b40bf as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/0647850e30f64b9489f501cf0d5b40bf 2023-03-31 16:55:30,277 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2e08a42276d589261ea162f9850b0f0a/B of 2e08a42276d589261ea162f9850b0f0a into 0647850e30f64b9489f501cf0d5b40bf(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:55:30,277 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2e08a42276d589261ea162f9850b0f0a: 2023-03-31 16:55:30,277 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a., storeName=2e08a42276d589261ea162f9850b0f0a/B, priority=13, startTime=1680281730219; duration=0sec 2023-03-31 16:55:30,277 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:55:30,277 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 16:55:30,278 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34524 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 16:55:30,278 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): 2e08a42276d589261ea162f9850b0f0a/C is initiating minor compaction (all files) 2023-03-31 16:55:30,279 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2e08a42276d589261ea162f9850b0f0a/C in TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:30,279 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/66504eb74c96489ab3110b8d4381fed3, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/ed4d02d4245e48bf9392e6059a997c7f, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/bf4b5703a27849f7abbf04a28df9a49c] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp, totalSize=33.7 K 2023-03-31 16:55:30,279 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 66504eb74c96489ab3110b8d4381fed3, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=273, earliestPutTs=1680281725160 2023-03-31 16:55:30,279 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting ed4d02d4245e48bf9392e6059a997c7f, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=280, earliestPutTs=1680281726317 2023-03-31 16:55:30,280 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting bf4b5703a27849f7abbf04a28df9a49c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=493, earliestPutTs=1680281728952 2023-03-31 16:55:30,289 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): 2e08a42276d589261ea162f9850b0f0a#C#compaction#388 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:55:30,349 DEBUG [Thread-2550] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:30,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:30,351 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:30,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:30,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:30,353 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:30,353 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:30,353 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:30,353 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:30,353 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:30,353 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:30,355 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:30,355 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:30,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:30,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:30,355 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:30,355 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:30,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:30,356 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:30,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:30,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:30,356 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:30,356 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:30,356 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:30,356 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:30,356 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:30,356 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:30,357 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:30,357 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:30,357 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:30,357 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. started... 2023-03-31 16:55:30,357 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2e08a42276d589261ea162f9850b0f0a 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-03-31 16:55:30,357 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e08a42276d589261ea162f9850b0f0a, store=A 2023-03-31 16:55:30,357 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:30,357 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e08a42276d589261ea162f9850b0f0a, store=B 2023-03-31 16:55:30,357 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:30,357 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e08a42276d589261ea162f9850b0f0a, store=C 2023-03-31 16:55:30,357 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:30,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on 2e08a42276d589261ea162f9850b0f0a 2023-03-31 16:55:30,524 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. as already flushing 2023-03-31 16:55:30,573 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680281730551, max timestamp=1680281730571Num uniques 859; 2023-03-31 16:55:30,573 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680281730551, max timestamp=1680281730571Num uniques 859; 2023-03-31 16:55:30,573 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 16:55:30,573 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 16:55:30,573 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680281730551, max timestamp=1680281730571Num uniques 859; 2023-03-31 16:55:30,573 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 16:55:30,623 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680281730599, max timestamp=1680281730620Num uniques 1018; 2023-03-31 16:55:30,623 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-03-31 16:55:30,623 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680281730599, max timestamp=1680281730620Num uniques 1018; 2023-03-31 16:55:30,624 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-03-31 16:55:30,623 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680281730599, max timestamp=1680281730620Num uniques 1018; 2023-03-31 16:55:30,624 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-03-31 16:55:30,685 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680281730661, max timestamp=1680281730682Num uniques 1077; 2023-03-31 16:55:30,685 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-03-31 16:55:30,685 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680281730661, max timestamp=1680281730682Num uniques 1077; 2023-03-31 16:55:30,685 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680281730661, max timestamp=1680281730682Num uniques 1077; 2023-03-31 16:55:30,685 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-03-31 16:55:30,685 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-03-31 16:55:30,708 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/C/954b9564506b4f5b88858d38b9b8804e as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/954b9564506b4f5b88858d38b9b8804e 2023-03-31 16:55:30,711 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2e08a42276d589261ea162f9850b0f0a/C of 2e08a42276d589261ea162f9850b0f0a into 954b9564506b4f5b88858d38b9b8804e(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:55:30,711 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2e08a42276d589261ea162f9850b0f0a: 2023-03-31 16:55:30,711 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a., storeName=2e08a42276d589261ea162f9850b0f0a/C, priority=13, startTime=1680281730219; duration=0sec 2023-03-31 16:55:30,711 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:55:30,730 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680281730719, max timestamp=1680281730728Num uniques 1086; 2023-03-31 16:55:30,730 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-03-31 16:55:30,730 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680281730719, max timestamp=1680281730728Num uniques 1086; 2023-03-31 16:55:30,731 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-03-31 16:55:30,731 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680281730719, max timestamp=1680281730728Num uniques 1086; 2023-03-31 16:55:30,731 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-03-31 16:55:30,768 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230331c801fe5bc51943c1881da19c323f5b75 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331c801fe5bc51943c1881da19c323f5b75 2023-03-31 16:55:30,788 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1680281730738, max timestamp=1680281730785Num uniques 1045; 2023-03-31 16:55:30,788 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-03-31 16:55:30,788 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1680281730738, max timestamp=1680281730785Num uniques 1045; 2023-03-31 16:55:30,788 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1680281730738, max timestamp=1680281730785Num uniques 1045; 2023-03-31 16:55:30,788 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-03-31 16:55:30,788 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-03-31 16:55:30,860 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:30,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53626 deadline: 1680281790853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:30,860 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1680281730844, max timestamp=1680281730858Num uniques 1104; 2023-03-31 16:55:30,860 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-03-31 16:55:30,860 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1680281730844, max timestamp=1680281730858Num uniques 1104; 2023-03-31 16:55:30,861 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1680281730844, max timestamp=1680281730858Num uniques 1104; 2023-03-31 16:55:30,861 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-03-31 16:55:30,861 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-03-31 16:55:30,939 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1680281730897, max timestamp=1680281730936Num uniques 1013; 2023-03-31 16:55:30,939 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-03-31 16:55:30,939 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1680281730897, max timestamp=1680281730936Num uniques 1013; 2023-03-31 16:55:30,939 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-03-31 16:55:30,939 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1680281730897, max timestamp=1680281730936Num uniques 1013; 2023-03-31 16:55:30,939 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-03-31 16:55:31,017 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1680281730998, max timestamp=1680281731014Num uniques 1022; 2023-03-31 16:55:31,017 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1680281730998, max timestamp=1680281731014Num uniques 1022; 2023-03-31 16:55:31,017 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-03-31 16:55:31,017 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1680281730998, max timestamp=1680281731014Num uniques 1022; 2023-03-31 16:55:31,017 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-03-31 16:55:31,017 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-03-31 16:55:31,074 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1680281731060, max timestamp=1680281731071Num uniques 1031; 2023-03-31 16:55:31,074 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-03-31 16:55:31,074 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1680281731060, max timestamp=1680281731071Num uniques 1031; 2023-03-31 16:55:31,074 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-03-31 16:55:31,074 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1680281731060, max timestamp=1680281731071Num uniques 1031; 2023-03-31 16:55:31,074 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-03-31 16:55:31,083 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/A/87b9ebcb1ce0494981e9bc97efe3854d as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/87b9ebcb1ce0494981e9bc97efe3854d 2023-03-31 16:55:31,087 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2e08a42276d589261ea162f9850b0f0a/A of 2e08a42276d589261ea162f9850b0f0a into 87b9ebcb1ce0494981e9bc97efe3854d(size=25.9 K), total size for store is 25.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:55:31,087 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2e08a42276d589261ea162f9850b0f0a: 2023-03-31 16:55:31,087 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a., storeName=2e08a42276d589261ea162f9850b0f0a/A, priority=13, startTime=1680281730219; duration=0sec 2023-03-31 16:55:31,087 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:55:31,112 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=340, cellSize=15570, totalHeapSize=35600, min timestamp=1680281731096, max timestamp=1680281731108Num uniques 1040; 2023-03-31 16:55:31,112 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=340, cellSize=15570, totalHeapSize=35600, min timestamp=1680281731096, max timestamp=1680281731108Num uniques 1040; 2023-03-31 16:55:31,112 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=340, cellSize=15570, totalHeapSize=35600, min timestamp=1680281731096, max timestamp=1680281731108Num uniques 1040; 2023-03-31 16:55:31,112 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=15570, suffix heap size=136128,new segment heap size=35600 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=340 2023-03-31 16:55:31,112 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=15570, suffix heap size=136128,new segment heap size=35600 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=340 2023-03-31 16:55:31,112 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=15570, suffix heap size=136128,new segment heap size=35600 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=340 2023-03-31 16:55:31,142 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:31,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53640 deadline: 1680281791135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:31,143 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1680281731133, max timestamp=1680281731141Num uniques 1199; 2023-03-31 16:55:31,143 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=54914, new segment data size=11404, suffix heap size=152664,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1199, new segment cells count=249 2023-03-31 16:55:31,144 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1680281731133, max timestamp=1680281731141Num uniques 1199; 2023-03-31 16:55:31,144 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=54914, new segment data size=11404, suffix heap size=152664,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1199, new segment cells count=249 2023-03-31 16:55:31,144 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1680281731133, max timestamp=1680281731141Num uniques 1199; 2023-03-31 16:55:31,144 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=54914, new segment data size=11404, suffix heap size=152664,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1199, new segment cells count=249 2023-03-31 16:55:31,173 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=501, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/A/02b785a12c57488aa86cfbedc928c28f 2023-03-31 16:55:31,176 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:31,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53680 deadline: 1680281791171, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:31,177 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1680281731164, max timestamp=1680281731175Num uniques 1108; 2023-03-31 16:55:31,177 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=7230, suffix heap size=143200,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=158 2023-03-31 16:55:31,178 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1680281731164, max timestamp=1680281731175Num uniques 1108; 2023-03-31 16:55:31,178 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=258, cellSize=11810, totalHeapSize=27072, min timestamp=1680281731164, max timestamp=1680281731175Num uniques 1108; 2023-03-31 16:55:31,178 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=7230, suffix heap size=143200,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=158 2023-03-31 16:55:31,178 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=11810, suffix heap size=143200,new segment heap size=27072 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=258 2023-03-31 16:55:31,213 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=267, cellSize=12222, totalHeapSize=28008, min timestamp=1680281731193, max timestamp=1680281731211Num uniques 1117; 2023-03-31 16:55:31,213 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=51152, new segment data size=12222, suffix heap size=144136,new segment heap size=28008 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1117, new segment cells count=267 2023-03-31 16:55:31,213 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=267, cellSize=12222, totalHeapSize=28008, min timestamp=1680281731193, max timestamp=1680281731211Num uniques 1017; 2023-03-31 16:55:31,213 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=267, cellSize=12222, totalHeapSize=28008, min timestamp=1680281731193, max timestamp=1680281731211Num uniques 1017; 2023-03-31 16:55:31,214 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=12222, suffix heap size=133736,new segment heap size=28008 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=267 2023-03-31 16:55:31,214 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=12222, suffix heap size=133736,new segment heap size=28008 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=267 2023-03-31 16:55:31,244 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=276, cellSize=12636, totalHeapSize=28944, min timestamp=1680281731235, max timestamp=1680281731242Num uniques 1126; 2023-03-31 16:55:31,244 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=51566, new segment data size=12636, suffix heap size=145072,new segment heap size=28944 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1126, new segment cells count=276 2023-03-31 16:55:31,245 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=276, cellSize=12636, totalHeapSize=28944, min timestamp=1680281731235, max timestamp=1680281731242Num uniques 1126; 2023-03-31 16:55:31,245 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=51566, new segment data size=12636, suffix heap size=145072,new segment heap size=28944 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1126, new segment cells count=276 2023-03-31 16:55:31,245 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=276, cellSize=12636, totalHeapSize=28944, min timestamp=1680281731235, max timestamp=1680281731242Num uniques 1126; 2023-03-31 16:55:31,245 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=51566, new segment data size=12636, suffix heap size=145072,new segment heap size=28944 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1126, new segment cells count=276 2023-03-31 16:55:31,274 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:31,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53692 deadline: 1680281791271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:31,275 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=285, cellSize=13050, totalHeapSize=29880, min timestamp=1680281731260, max timestamp=1680281731273Num uniques 1135; 2023-03-31 16:55:31,275 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=51980, new segment data size=13050, suffix heap size=146008,new segment heap size=29880 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1135, new segment cells count=285 2023-03-31 16:55:31,275 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=285, cellSize=13050, totalHeapSize=29880, min timestamp=1680281731260, max timestamp=1680281731273Num uniques 1135; 2023-03-31 16:55:31,275 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=285, cellSize=13050, totalHeapSize=29880, min timestamp=1680281731260, max timestamp=1680281731273Num uniques 1135; 2023-03-31 16:55:31,275 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=51980, new segment data size=13050, suffix heap size=146008,new segment heap size=29880 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1135, new segment cells count=285 2023-03-31 16:55:31,275 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=51980, new segment data size=13050, suffix heap size=146008,new segment heap size=29880 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1135, new segment cells count=285 2023-03-31 16:55:31,306 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=244, cellSize=11174, totalHeapSize=25616, min timestamp=1680281731294, max timestamp=1680281731304Num uniques 1144; 2023-03-31 16:55:31,306 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=52394, new segment data size=11174, suffix heap size=146944,new segment heap size=25616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1144, new segment cells count=244 2023-03-31 16:55:31,307 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=244, cellSize=11174, totalHeapSize=25616, min timestamp=1680281731294, max timestamp=1680281731304Num uniques 1144; 2023-03-31 16:55:31,307 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=52394, new segment data size=11174, suffix heap size=146944,new segment heap size=25616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1144, new segment cells count=244 2023-03-31 16:55:31,307 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=244, cellSize=11174, totalHeapSize=25616, min timestamp=1680281731294, max timestamp=1680281731304Num uniques 1144; 2023-03-31 16:55:31,307 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=52394, new segment data size=11174, suffix heap size=146944,new segment heap size=25616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1144, new segment cells count=244 2023-03-31 16:55:31,342 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=353, cellSize=16165, totalHeapSize=36952, min timestamp=1680281731317, max timestamp=1680281731340Num uniques 1103; 2023-03-31 16:55:31,342 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=353, cellSize=16165, totalHeapSize=36952, min timestamp=1680281731317, max timestamp=1680281731340Num uniques 1103; 2023-03-31 16:55:31,342 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50515, new segment data size=16165, suffix heap size=142680,new segment heap size=36952 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1103, new segment cells count=353 2023-03-31 16:55:31,342 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50515, new segment data size=16165, suffix heap size=142680,new segment heap size=36952 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1103, new segment cells count=353 2023-03-31 16:55:31,343 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=353, cellSize=16165, totalHeapSize=36952, min timestamp=1680281731317, max timestamp=1680281731340Num uniques 1103; 2023-03-31 16:55:31,343 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50515, new segment data size=16165, suffix heap size=142680,new segment heap size=36952 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1103, new segment cells count=353 2023-03-31 16:55:31,367 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:31,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53652 deadline: 1680281791365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:31,368 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=262, cellSize=11992, totalHeapSize=27488, min timestamp=1680281731358, max timestamp=1680281731366Num uniques 1212; 2023-03-31 16:55:31,368 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=55502, new segment data size=11992, suffix heap size=154016,new segment heap size=27488 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1212, new segment cells count=262 2023-03-31 16:55:31,368 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=262, cellSize=11992, totalHeapSize=27488, min timestamp=1680281731358, max timestamp=1680281731366Num uniques 1212; 2023-03-31 16:55:31,368 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=55502, new segment data size=11992, suffix heap size=154016,new segment heap size=27488 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1212, new segment cells count=262 2023-03-31 16:55:31,369 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=262, cellSize=11992, totalHeapSize=27488, min timestamp=1680281731358, max timestamp=1680281731366Num uniques 1212; 2023-03-31 16:55:31,369 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=55502, new segment data size=11992, suffix heap size=154016,new segment heap size=27488 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1212, new segment cells count=262 2023-03-31 16:55:31,403 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:31,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53626 deadline: 1680281791401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:31,404 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=321, cellSize=14696, totalHeapSize=33624, min timestamp=1680281731392, max timestamp=1680281731402Num uniques 1121; 2023-03-31 16:55:31,404 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=51336, new segment data size=14696, suffix heap size=144552,new segment heap size=33624 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1121, new segment cells count=321 2023-03-31 16:55:31,405 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=321, cellSize=14696, totalHeapSize=33624, min timestamp=1680281731392, max timestamp=1680281731402Num uniques 1121; 2023-03-31 16:55:31,405 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=51336, new segment data size=14696, suffix heap size=144552,new segment heap size=33624 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1121, new segment cells count=321 2023-03-31 16:55:31,405 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=321, cellSize=14696, totalHeapSize=33624, min timestamp=1680281731392, max timestamp=1680281731402Num uniques 1121; 2023-03-31 16:55:31,405 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=51336, new segment data size=14696, suffix heap size=144552,new segment heap size=33624 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1121, new segment cells count=321 2023-03-31 16:55:31,439 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=280, cellSize=12820, totalHeapSize=29360, min timestamp=1680281731428, max timestamp=1680281731437Num uniques 1180; 2023-03-31 16:55:31,439 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=54040, new segment data size=12820, suffix heap size=150688,new segment heap size=29360 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1180, new segment cells count=280 2023-03-31 16:55:31,440 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=280, cellSize=12820, totalHeapSize=29360, min timestamp=1680281731428, max timestamp=1680281731437Num uniques 1180; 2023-03-31 16:55:31,440 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=54040, new segment data size=12820, suffix heap size=150688,new segment heap size=29360 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1180, new segment cells count=280 2023-03-31 16:55:31,440 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=280, cellSize=12820, totalHeapSize=29360, min timestamp=1680281731428, max timestamp=1680281731437Num uniques 1180; 2023-03-31 16:55:31,440 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=54040, new segment data size=12820, suffix heap size=150688,new segment heap size=29360 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1180, new segment cells count=280 2023-03-31 16:55:31,472 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:31,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53652 deadline: 1680281791468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:31,474 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=289, cellSize=13234, totalHeapSize=30296, min timestamp=1680281731455, max timestamp=1680281731471Num uniques 1139; 2023-03-31 16:55:31,474 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=289, cellSize=13234, totalHeapSize=30296, min timestamp=1680281731455, max timestamp=1680281731471Num uniques 1139; 2023-03-31 16:55:31,474 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=289, cellSize=13234, totalHeapSize=30296, min timestamp=1680281731455, max timestamp=1680281731471Num uniques 1139; 2023-03-31 16:55:31,474 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=52164, new segment data size=13234, suffix heap size=146424,new segment heap size=30296 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1139, new segment cells count=289 2023-03-31 16:55:31,474 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=52164, new segment data size=13234, suffix heap size=146424,new segment heap size=30296 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1139, new segment cells count=289 2023-03-31 16:55:31,474 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=52164, new segment data size=13234, suffix heap size=146424,new segment heap size=30296 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1139, new segment cells count=289 2023-03-31 16:55:31,506 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=248, cellSize=11358, totalHeapSize=26032, min timestamp=1680281731489, max timestamp=1680281731504Num uniques 1148; 2023-03-31 16:55:31,506 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=52578, new segment data size=11358, suffix heap size=147360,new segment heap size=26032 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1148, new segment cells count=248 2023-03-31 16:55:31,507 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=248, cellSize=11358, totalHeapSize=26032, min timestamp=1680281731489, max timestamp=1680281731504Num uniques 1148; 2023-03-31 16:55:31,507 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=52578, new segment data size=11358, suffix heap size=147360,new segment heap size=26032 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1148, new segment cells count=248 2023-03-31 16:55:31,507 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=248, cellSize=11358, totalHeapSize=26032, min timestamp=1680281731489, max timestamp=1680281731504Num uniques 1148; 2023-03-31 16:55:31,507 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=52578, new segment data size=11358, suffix heap size=147360,new segment heap size=26032 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1148, new segment cells count=248 2023-03-31 16:55:31,536 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:31,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53626 deadline: 1680281791530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:31,537 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=257, cellSize=11765, totalHeapSize=26968, min timestamp=1680281731522, max timestamp=1680281731535Num uniques 1107; 2023-03-31 16:55:31,537 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=257, cellSize=11765, totalHeapSize=26968, min timestamp=1680281731522, max timestamp=1680281731535Num uniques 1107; 2023-03-31 16:55:31,538 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=257, cellSize=11765, totalHeapSize=26968, min timestamp=1680281731522, max timestamp=1680281731535Num uniques 1107; 2023-03-31 16:55:31,538 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50695, new segment data size=11765, suffix heap size=143096,new segment heap size=26968 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1107, new segment cells count=257 2023-03-31 16:55:31,538 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50695, new segment data size=11765, suffix heap size=143096,new segment heap size=26968 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1107, new segment cells count=257 2023-03-31 16:55:31,538 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50695, new segment data size=11765, suffix heap size=143096,new segment heap size=26968 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1107, new segment cells count=257 2023-03-31 16:55:31,584 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=501 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/B/9120096c80b84934930ca22943118bf4 2023-03-31 16:55:31,593 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=501 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/C/7dda3dc51f514d4bb1d2d3b00e0ea79a 2023-03-31 16:55:31,597 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/A/02b785a12c57488aa86cfbedc928c28f as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/02b785a12c57488aa86cfbedc928c28f 2023-03-31 16:55:31,601 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/02b785a12c57488aa86cfbedc928c28f, entries=150, sequenceid=501, filesize=25.4 K 2023-03-31 16:55:31,601 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/B/9120096c80b84934930ca22943118bf4 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/9120096c80b84934930ca22943118bf4 2023-03-31 16:55:31,602 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:31,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53640 deadline: 1680281791596, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:31,602 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:31,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53680 deadline: 1680281791599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:31,603 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=166, cellSize=7596, totalHeapSize=17504, min timestamp=1680281731593, max timestamp=1680281731601Num uniques 1116; 2023-03-31 16:55:31,603 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=166, cellSize=7596, totalHeapSize=17504, min timestamp=1680281731593, max timestamp=1680281731601Num uniques 1116; 2023-03-31 16:55:31,603 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=51106, new segment data size=7596, suffix heap size=144032,new segment heap size=17504 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1116, new segment cells count=166 2023-03-31 16:55:31,603 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=51106, new segment data size=7596, suffix heap size=144032,new segment heap size=17504 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1116, new segment cells count=166 2023-03-31 16:55:31,604 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=166, cellSize=7596, totalHeapSize=17504, min timestamp=1680281731593, max timestamp=1680281731601Num uniques 1116; 2023-03-31 16:55:31,604 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=51106, new segment data size=7596, suffix heap size=144032,new segment heap size=17504 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1116, new segment cells count=166 2023-03-31 16:55:31,610 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/9120096c80b84934930ca22943118bf4, entries=150, sequenceid=501, filesize=11.9 K 2023-03-31 16:55:31,611 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/C/7dda3dc51f514d4bb1d2d3b00e0ea79a as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/7dda3dc51f514d4bb1d2d3b00e0ea79a 2023-03-31 16:55:31,614 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/7dda3dc51f514d4bb1d2d3b00e0ea79a, entries=150, sequenceid=501, filesize=11.9 K 2023-03-31 16:55:31,614 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=40.25 KB/41220 for 2e08a42276d589261ea162f9850b0f0a in 1257ms, sequenceid=501, compaction requested=false 2023-03-31 16:55:31,614 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2e08a42276d589261ea162f9850b0f0a: 2023-03-31 16:55:31,614 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:31,614 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:31,615 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:31,615 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:31,615 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:31,615 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:31,616 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:31,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:31,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:31,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:31,616 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:31,616 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:31,617 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:31,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:31,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:31,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:31,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:31,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:31,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:31,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53f02f98[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:31,618 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:31,618 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:31,619 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:31,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:31,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:31,619 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:31,619 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:31,619 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:31,619 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:31,619 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:31,621 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:31,621 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:31,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:31,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:31,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:31,621 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:31,622 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:31,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:31,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:31,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:31,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:31,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:31,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:31,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:31,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:31,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:31,623 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:31,623 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:31,623 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:31,623 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:31,624 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:31,625 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:31,625 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:31,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:31,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:31,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:31,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:31,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:31,625 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:31,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on 2e08a42276d589261ea162f9850b0f0a 2023-03-31 16:55:31,625 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2e08a42276d589261ea162f9850b0f0a 3/3 column families, dataSize=53.67 KB heapSize=145.27 KB 2023-03-31 16:55:31,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:31,626 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e08a42276d589261ea162f9850b0f0a, store=A 2023-03-31 16:55:31,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:31,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:31,626 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:31,626 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:55:31,626 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e08a42276d589261ea162f9850b0f0a, store=B 2023-03-31 16:55:31,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:31,626 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:55:31,626 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e08a42276d589261ea162f9850b0f0a, store=C 2023-03-31 16:55:31,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:31,626 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:55:31,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:31,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:31,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:31,627 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:31,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:31,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:31,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:31,630 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:31,630 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:31,630 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:31,630 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:31,630 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:31,630 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:31,630 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:31,631 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:31,631 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:31,631 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:31,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:31,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:31,631 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:31,631 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:31,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:31,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:31,632 DEBUG [Thread-2550] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:31,632 DEBUG [Thread-2550] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:31,640 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303311b036c53697d4ac7838935b3978053f1 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303311b036c53697d4ac7838935b3978053f1 2023-03-31 16:55:31,689 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680281731671, max timestamp=1680281731687Num uniques 859; 2023-03-31 16:55:31,689 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680281731671, max timestamp=1680281731687Num uniques 859; 2023-03-31 16:55:31,689 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680281731671, max timestamp=1680281731687Num uniques 859; 2023-03-31 16:55:31,689 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 16:55:31,689 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 16:55:31,689 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 16:55:31,722 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:31,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53640 deadline: 1680281791711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:31,722 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:31,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53652 deadline: 1680281791717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:31,731 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:31,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53692 deadline: 1680281791719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:31,731 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:31,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53626 deadline: 1680281791720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:31,731 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:31,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53680 deadline: 1680281791721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:31,732 DEBUG [Thread-2550] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:31,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:31,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:31,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:31,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:31,735 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:31,735 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:31,735 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:31,735 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:31,735 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:31,735 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:31,736 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:31,736 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:31,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:31,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:31,737 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:31,737 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:31,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:31,737 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:31,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:31,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:31,737 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:31,737 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:31,737 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:31,738 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:31,738 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:31,738 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:31,738 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:31,738 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:31,738 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:31,738 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. started... 2023-03-31 16:55:31,738 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. as already flushing 2023-03-31 16:55:31,738 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:31,738 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:31,738 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:31,738 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:31,738 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:31,738 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:31,744 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:31,744 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:31,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:31,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:31,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:31,744 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:31,744 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:31,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:31,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:31,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:31,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:31,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:31,745 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:31,745 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e050d96[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:31,745 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:31,745 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:31,746 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:31,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:31,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:31,746 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:31,747 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:31,747 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:31,747 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:31,747 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:31,748 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:31,748 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:31,748 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:31,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:31,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:31,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:31,748 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:31,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:31,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:31,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:31,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:31,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:31,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:31,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:31,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:31,751 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:31,751 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:31,751 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:31,751 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:31,751 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:31,751 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:31,752 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:31,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:31,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:31,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:31,752 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:31,752 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:31,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:31,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:31,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:31,753 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:31,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:31,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:31,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:31,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:31,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:31,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:31,754 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:31,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:31,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:31,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:31,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:31,757 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:31,757 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:31,757 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:31,757 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:31,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:31,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:31,757 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:31,757 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:31,757 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:31,757 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:31,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:31,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:31,757 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:31,757 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:31,757 DEBUG [Thread-2550] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:31,758 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:31,758 DEBUG [Thread-2550] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:31,758 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:31,758 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:31,825 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:31,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53640 deadline: 1680281791823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:31,826 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:31,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53652 deadline: 1680281791823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:31,838 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:31,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53692 deadline: 1680281791832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:31,838 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:31,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53626 deadline: 1680281791832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:31,838 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:31,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 279 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53680 deadline: 1680281791832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:31,858 DEBUG [Thread-2550] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:31,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:31,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:31,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:31,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:31,861 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:31,861 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:31,861 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:31,861 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:31,861 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:31,861 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:31,862 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:31,863 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:31,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:31,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:31,864 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:31,864 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:31,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:31,864 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:31,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:31,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:31,865 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:31,865 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:31,865 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:31,865 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:31,865 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:31,865 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:31,865 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:31,865 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:31,865 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:31,865 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. started... 2023-03-31 16:55:31,865 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. as already flushing 2023-03-31 16:55:31,865 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:31,865 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:31,865 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:31,866 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:31,866 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:31,866 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:31,867 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:31,867 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:31,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:31,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:31,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:31,867 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:31,867 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:31,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:31,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:31,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:31,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:31,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:31,868 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:31,868 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f68e1d0[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:31,868 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:31,868 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:31,870 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:31,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:31,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:31,870 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:31,870 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:31,870 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:31,870 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:31,870 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:31,872 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:31,872 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:31,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:31,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:31,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:31,872 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:31,872 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:31,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:31,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:31,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:31,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:31,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:31,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:31,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:31,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:31,875 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:31,875 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:31,875 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:31,875 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:31,875 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:31,875 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:31,877 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:31,877 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:31,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:31,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:31,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:31,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:31,877 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:31,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:31,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:31,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:31,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:31,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:31,877 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:31,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:31,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:31,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:31,878 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:31,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:31,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:31,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:31,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:31,880 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:31,880 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:31,880 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:31,880 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:31,880 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:31,880 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:31,880 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:31,880 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:31,880 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:31,881 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:31,881 DEBUG [Thread-2550] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:31,881 DEBUG [Thread-2550] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:31,881 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:31,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:31,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:31,881 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:31,881 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:31,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:31,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:31,981 DEBUG [Thread-2550] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:31,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:31,983 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:31,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:31,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:31,984 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:31,984 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:31,984 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:31,984 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:31,985 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:31,985 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:31,986 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:31,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:31,986 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:31,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:31,986 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:31,986 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:31,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:31,987 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:31,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:31,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:31,987 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:31,987 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:31,987 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:31,987 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:31,988 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:31,988 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:31,988 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:31,988 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:31,988 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:31,988 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. started... 2023-03-31 16:55:31,988 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. as already flushing 2023-03-31 16:55:31,988 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:31,988 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:31,988 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:31,988 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:31,988 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:31,988 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:31,990 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:31,990 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:31,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:31,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:31,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:31,991 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:31,991 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:31,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:31,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:31,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:31,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:31,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:31,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:31,993 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:31,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@254ae880[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:31,993 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:31,994 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:31,994 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:31,995 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:31,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:31,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:31,995 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:31,995 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:31,995 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:31,996 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:31,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:31,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:31,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:31,997 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:31,997 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:31,997 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:31,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:31,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:31,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:31,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:31,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:31,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:31,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:31,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:31,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:31,999 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:31,999 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:31,999 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:31,999 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:31,999 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:32,000 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:32,001 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:32,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:32,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:32,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:32,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:32,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:32,001 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:32,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:32,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:32,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:32,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:32,001 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:32,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:32,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:32,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:32,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:32,002 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:32,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:32,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:32,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:32,005 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:32,005 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:32,005 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:32,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:32,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:32,005 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:32,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:32,006 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:32,005 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:32,005 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:32,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:32,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:32,006 DEBUG [Thread-2550] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:32,006 DEBUG [Thread-2550] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:32,006 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:32,006 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:32,006 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:32,006 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:32,032 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:32,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53640 deadline: 1680281792026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:32,032 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:32,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53652 deadline: 1680281792027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:32,040 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:32,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 276 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53692 deadline: 1680281792039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:32,045 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=923, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/A/94c85cba95c14b7dbedf2288e21d394a 2023-03-31 16:55:32,046 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:32,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53626 deadline: 1680281792040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:32,046 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:32,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 281 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53680 deadline: 1680281792040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:32,106 DEBUG [Thread-2550] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:32,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:32,110 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:32,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:32,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:32,111 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:32,111 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:32,111 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:32,111 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:32,112 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:32,112 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:32,113 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:32,113 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:32,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:32,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:32,113 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:32,114 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:32,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:32,114 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:32,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:32,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:32,114 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:32,114 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:32,114 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:32,115 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:32,115 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:32,115 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:32,115 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:32,115 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:32,115 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:32,115 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. started... 2023-03-31 16:55:32,115 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. as already flushing 2023-03-31 16:55:32,115 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:32,115 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:32,115 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:32,115 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:32,115 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:32,115 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:32,117 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:32,117 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:32,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:32,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:32,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:32,117 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:32,117 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:32,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:32,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:32,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:32,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:32,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:32,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:32,119 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:32,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77149f5e[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:32,119 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:32,120 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:32,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:32,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:32,120 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:32,121 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:32,121 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:32,120 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:32,121 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:32,126 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:32,126 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:32,126 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:32,126 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:32,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:32,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:32,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:32,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:32,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:32,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:32,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:32,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:32,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:32,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:32,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:32,128 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:32,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:32,128 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:32,128 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:32,128 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:32,128 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:32,130 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:32,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:32,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:32,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:32,130 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:32,130 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:32,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:32,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:32,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:32,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:32,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:32,131 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:32,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:32,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:32,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:32,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:32,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:32,132 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:32,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:32,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:32,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:32,135 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:32,135 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:32,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:32,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:32,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:32,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:32,135 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:32,135 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:32,135 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:32,135 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:32,135 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:32,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:32,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:32,135 DEBUG [Thread-2550] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:32,135 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:32,135 DEBUG [Thread-2550] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:32,135 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:32,135 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:32,235 DEBUG [Thread-2550] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:32,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:32,244 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:32,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:32,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:32,245 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:32,245 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:32,245 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:32,245 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:32,246 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:32,246 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:32,247 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:32,247 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:32,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:32,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:32,247 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:32,247 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:32,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:32,248 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:32,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:32,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:32,248 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:32,248 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:32,248 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:32,248 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:32,248 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:32,248 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:32,249 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:32,249 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:32,249 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:32,249 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. started... 2023-03-31 16:55:32,249 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. as already flushing 2023-03-31 16:55:32,249 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:32,249 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:32,249 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:32,249 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:32,249 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:32,249 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:32,254 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:32,254 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:32,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:32,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:32,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:32,254 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:32,254 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:32,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:32,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:32,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:32,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:32,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:32,256 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:32,256 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@414b4597[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:32,256 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:32,256 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:32,258 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:32,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:32,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:32,258 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:32,258 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:32,258 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:32,258 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:32,258 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:32,259 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:32,259 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:32,259 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:32,259 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:32,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:32,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:32,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:32,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:32,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:32,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:32,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:32,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:32,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:32,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:32,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:32,262 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:32,262 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:32,262 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:32,262 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:32,262 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:32,262 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:32,264 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:32,264 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:32,264 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:32,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:32,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:32,264 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:32,265 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:32,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:32,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:32,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:32,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:32,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:32,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:32,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:32,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:32,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:32,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:32,269 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:32,269 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:32,269 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:32,269 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:32,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:32,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:32,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:32,269 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:32,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:32,269 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:32,269 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:32,269 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:32,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:32,269 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:32,270 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:32,269 DEBUG [Thread-2550] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:32,270 DEBUG [Thread-2550] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:32,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:32,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:32,270 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:32,339 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:32,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53652 deadline: 1680281792335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:32,340 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:32,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53640 deadline: 1680281792335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:32,347 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:32,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 278 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53692 deadline: 1680281792342, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:32,348 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:32,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53626 deadline: 1680281792348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:32,349 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:32,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 283 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53680 deadline: 1680281792348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:32,370 DEBUG [Thread-2550] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:32,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:32,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:32,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:32,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:32,374 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:32,374 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:32,374 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:32,374 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:32,374 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:32,374 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:32,376 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:32,376 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:32,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:32,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:32,376 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:32,376 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:32,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:32,376 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:32,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:32,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:32,377 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:32,377 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:32,377 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:32,377 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:32,377 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:32,377 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:32,377 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:32,377 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:32,377 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:32,377 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. started... 2023-03-31 16:55:32,377 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. as already flushing 2023-03-31 16:55:32,378 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:32,378 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:32,378 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:32,378 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:32,378 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:32,378 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:32,379 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:32,379 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:32,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:32,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:32,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:32,379 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:32,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:32,380 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:32,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:32,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:32,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:32,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:32,381 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:32,381 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9654a0f[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:32,381 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:32,381 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:32,382 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:32,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:32,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:32,382 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:32,382 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:32,382 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:32,382 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:32,382 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:32,384 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:32,384 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:32,384 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:32,384 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:32,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:32,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:32,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:32,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:32,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:32,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:32,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:32,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:32,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:32,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:32,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:32,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:32,387 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:32,387 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:32,387 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:32,387 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:32,387 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:32,389 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:32,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:32,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:32,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:32,389 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:32,389 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:32,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:32,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:32,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:32,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:32,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:32,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:32,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:32,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:32,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:32,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:32,391 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:32,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:32,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:32,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:32,392 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:32,394 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:32,394 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:32,394 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:32,394 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:32,394 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:32,394 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:32,394 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:32,394 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:32,394 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:32,394 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:32,394 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:32,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:32,394 DEBUG [Thread-2550] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:32,394 DEBUG [Thread-2550] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:32,394 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:32,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:32,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:32,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:32,454 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=923 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/B/f662a530ec434af89b3591078d3b62d6 2023-03-31 16:55:32,464 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=923 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/C/73bac03200104ecaad413793455e2ad6 2023-03-31 16:55:32,467 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/A/94c85cba95c14b7dbedf2288e21d394a as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/94c85cba95c14b7dbedf2288e21d394a 2023-03-31 16:55:32,470 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/94c85cba95c14b7dbedf2288e21d394a, entries=200, sequenceid=923, filesize=32.3 K 2023-03-31 16:55:32,471 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/B/f662a530ec434af89b3591078d3b62d6 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/f662a530ec434af89b3591078d3b62d6 2023-03-31 16:55:32,473 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/f662a530ec434af89b3591078d3b62d6, entries=150, sequenceid=923, filesize=11.9 K 2023-03-31 16:55:32,474 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/C/73bac03200104ecaad413793455e2ad6 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/73bac03200104ecaad413793455e2ad6 2023-03-31 16:55:32,477 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/73bac03200104ecaad413793455e2ad6, entries=150, sequenceid=923, filesize=11.9 K 2023-03-31 16:55:32,477 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~145.22 KB/148704, currentSize=131.94 KB/135110 for 2e08a42276d589261ea162f9850b0f0a in 852ms, sequenceid=923, compaction requested=true 2023-03-31 16:55:32,477 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2e08a42276d589261ea162f9850b0f0a: 2023-03-31 16:55:32,478 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:55:32,478 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 16:55:32,478 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 16:55:32,478 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:55:32,478 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:55:32,479 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85690 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 16:55:32,479 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37070 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 16:55:32,479 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1912): 2e08a42276d589261ea162f9850b0f0a/A is initiating minor compaction (all files) 2023-03-31 16:55:32,479 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): 2e08a42276d589261ea162f9850b0f0a/B is initiating minor compaction (all files) 2023-03-31 16:55:32,479 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2e08a42276d589261ea162f9850b0f0a/A in TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:32,479 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2e08a42276d589261ea162f9850b0f0a/B in TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:32,479 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/87b9ebcb1ce0494981e9bc97efe3854d, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/02b785a12c57488aa86cfbedc928c28f, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/94c85cba95c14b7dbedf2288e21d394a] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp, totalSize=83.7 K 2023-03-31 16:55:32,479 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/0647850e30f64b9489f501cf0d5b40bf, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/9120096c80b84934930ca22943118bf4, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/f662a530ec434af89b3591078d3b62d6] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp, totalSize=36.2 K 2023-03-31 16:55:32,480 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 87b9ebcb1ce0494981e9bc97efe3854d, keycount=150, bloomtype=ROW, size=25.9 K, encoding=NONE, compression=NONE, seqNum=493, earliestPutTs=1680281728952 2023-03-31 16:55:32,480 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 0647850e30f64b9489f501cf0d5b40bf, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=493, earliestPutTs=1680281728952 2023-03-31 16:55:32,480 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 02b785a12c57488aa86cfbedc928c28f, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=501, earliestPutTs=1680281729362 2023-03-31 16:55:32,480 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 9120096c80b84934930ca22943118bf4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=501, earliestPutTs=1680281729362 2023-03-31 16:55:32,481 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 94c85cba95c14b7dbedf2288e21d394a, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=923, earliestPutTs=1680281731594 2023-03-31 16:55:32,481 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting f662a530ec434af89b3591078d3b62d6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=923, earliestPutTs=1680281731594 2023-03-31 16:55:32,489 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): 2e08a42276d589261ea162f9850b0f0a#B#compaction#396 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:55:32,494 DEBUG [Thread-2550] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:32,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:32,497 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2e08a42276d589261ea162f9850b0f0a#A#compaction#395 average throughput is 2.19 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:55:32,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:32,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:32,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:32,504 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:32,504 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:32,504 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:32,504 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:32,505 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:32,505 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:32,506 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:32,506 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:32,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:32,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:32,506 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:32,506 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:32,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:32,506 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:32,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:32,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:32,507 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:32,515 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:32,515 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:32,515 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:32,515 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:32,515 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:32,520 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:32,520 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:32,520 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:32,520 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. started... 2023-03-31 16:55:32,520 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2e08a42276d589261ea162f9850b0f0a 3/3 column families, dataSize=131.94 KB heapSize=375.22 KB 2023-03-31 16:55:32,520 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e08a42276d589261ea162f9850b0f0a, store=A 2023-03-31 16:55:32,521 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:55:32,521 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e08a42276d589261ea162f9850b0f0a, store=B 2023-03-31 16:55:32,521 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:55:32,521 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e08a42276d589261ea162f9850b0f0a, store=C 2023-03-31 16:55:32,521 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:55:32,569 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/A/182e43fa1bfa4de78d67329ae8c08657 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/182e43fa1bfa4de78d67329ae8c08657 2023-03-31 16:55:32,581 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2e08a42276d589261ea162f9850b0f0a/A of 2e08a42276d589261ea162f9850b0f0a into 182e43fa1bfa4de78d67329ae8c08657(size=26.0 K), total size for store is 26.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:55:32,581 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2e08a42276d589261ea162f9850b0f0a: 2023-03-31 16:55:32,581 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a., storeName=2e08a42276d589261ea162f9850b0f0a/A, priority=13, startTime=1680281732478; duration=0sec 2023-03-31 16:55:32,581 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:55:32,581 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 16:55:32,582 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37070 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 16:55:32,582 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1912): 2e08a42276d589261ea162f9850b0f0a/C is initiating minor compaction (all files) 2023-03-31 16:55:32,582 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2e08a42276d589261ea162f9850b0f0a/C in TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:32,582 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/954b9564506b4f5b88858d38b9b8804e, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/7dda3dc51f514d4bb1d2d3b00e0ea79a, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/73bac03200104ecaad413793455e2ad6] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp, totalSize=36.2 K 2023-03-31 16:55:32,582 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 954b9564506b4f5b88858d38b9b8804e, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=493, earliestPutTs=1680281728952 2023-03-31 16:55:32,582 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 7dda3dc51f514d4bb1d2d3b00e0ea79a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=501, earliestPutTs=1680281729362 2023-03-31 16:55:32,583 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 73bac03200104ecaad413793455e2ad6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=923, earliestPutTs=1680281731594 2023-03-31 16:55:32,587 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2e08a42276d589261ea162f9850b0f0a#C#compaction#398 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:55:32,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on 2e08a42276d589261ea162f9850b0f0a 2023-03-31 16:55:32,849 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. as already flushing 2023-03-31 16:55:32,858 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:32,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 278 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53640 deadline: 1680281792855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:32,860 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:32,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53652 deadline: 1680281792856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:32,862 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:32,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 281 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53692 deadline: 1680281792857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:32,863 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:32,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53626 deadline: 1680281792858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:32,865 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:32,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 286 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53680 deadline: 1680281792858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:32,925 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/B/dd693f1f178544038668b80e543944ae as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/dd693f1f178544038668b80e543944ae 2023-03-31 16:55:32,928 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2e08a42276d589261ea162f9850b0f0a/B of 2e08a42276d589261ea162f9850b0f0a into dd693f1f178544038668b80e543944ae(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:55:32,929 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2e08a42276d589261ea162f9850b0f0a: 2023-03-31 16:55:32,929 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a., storeName=2e08a42276d589261ea162f9850b0f0a/B, priority=13, startTime=1680281732478; duration=0sec 2023-03-31 16:55:32,929 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:55:32,965 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230331b1386d3f7dc441eb88a1e4ccb4063787 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331b1386d3f7dc441eb88a1e4ccb4063787 2023-03-31 16:55:32,967 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:32,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 280 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53640 deadline: 1680281792959, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:32,969 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:32,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53652 deadline: 1680281792961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:32,969 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:32,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53626 deadline: 1680281792964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:32,972 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:32,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 283 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53692 deadline: 1680281792964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:32,972 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:32,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53680 deadline: 1680281792966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:32,980 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=959, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/A/39e7cce7987c4200b462277008145ca9 2023-03-31 16:55:32,998 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/C/4aebc16902d54bfa8cdbd41aaf534fac as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/4aebc16902d54bfa8cdbd41aaf534fac 2023-03-31 16:55:33,001 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2e08a42276d589261ea162f9850b0f0a/C of 2e08a42276d589261ea162f9850b0f0a into 4aebc16902d54bfa8cdbd41aaf534fac(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:55:33,001 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2e08a42276d589261ea162f9850b0f0a: 2023-03-31 16:55:33,001 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a., storeName=2e08a42276d589261ea162f9850b0f0a/C, priority=13, startTime=1680281732478; duration=0sec 2023-03-31 16:55:33,001 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:55:33,174 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:33,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 282 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53640 deadline: 1680281793169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:33,174 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:33,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53626 deadline: 1680281793170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:33,174 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:33,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53652 deadline: 1680281793171, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:33,176 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:33,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53680 deadline: 1680281793173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:33,176 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:33,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 285 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53692 deadline: 1680281793175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:33,394 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=959 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/B/234d0641c0ac4faf90398dcb73301abd 2023-03-31 16:55:33,404 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=959 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/C/e94917e437c14c8abd6fa91fcdcf3a08 2023-03-31 16:55:33,408 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/A/39e7cce7987c4200b462277008145ca9 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/39e7cce7987c4200b462277008145ca9 2023-03-31 16:55:33,411 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/39e7cce7987c4200b462277008145ca9, entries=150, sequenceid=959, filesize=25.4 K 2023-03-31 16:55:33,411 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/B/234d0641c0ac4faf90398dcb73301abd as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/234d0641c0ac4faf90398dcb73301abd 2023-03-31 16:55:33,415 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/234d0641c0ac4faf90398dcb73301abd, entries=150, sequenceid=959, filesize=11.9 K 2023-03-31 16:55:33,415 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/C/e94917e437c14c8abd6fa91fcdcf3a08 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/e94917e437c14c8abd6fa91fcdcf3a08 2023-03-31 16:55:33,418 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/e94917e437c14c8abd6fa91fcdcf3a08, entries=150, sequenceid=959, filesize=11.9 K 2023-03-31 16:55:33,419 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~131.94 KB/135110, heapSize ~375.17 KB/384176, currentSize=46.96 KB/48090 for 2e08a42276d589261ea162f9850b0f0a in 899ms, sequenceid=959, compaction requested=false 2023-03-31 16:55:33,419 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2e08a42276d589261ea162f9850b0f0a: 2023-03-31 16:55:33,419 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:33,419 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:33,419 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:33,419 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:33,419 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:33,419 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:33,421 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:33,421 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:33,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:33,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:33,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:33,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:33,422 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:33,422 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:33,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:33,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:33,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:33,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:33,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:33,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b99ad4b[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:33,423 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:33,423 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:33,425 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:33,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:33,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:33,425 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:33,425 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:33,425 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:33,426 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:33,426 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:33,427 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:33,427 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:33,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:33,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:33,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:33,427 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:33,427 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:33,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:33,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:33,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:33,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:33,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:33,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:33,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:33,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:33,431 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:33,431 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:33,431 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:33,431 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:33,431 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:33,431 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:33,436 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:33,436 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:33,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:33,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:33,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:33,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:33,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:33,436 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:33,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:33,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:33,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:33,437 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:33,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:33,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:33,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:33,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:33,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:33,438 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:33,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:33,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:33,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:33,440 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:33,440 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:33,440 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:33,440 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:33,440 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:33,440 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:33,440 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:33,440 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:33,440 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:33,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:33,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:33,441 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:33,441 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:33,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:33,441 DEBUG [Thread-2550] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:33,441 DEBUG [Thread-2550] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:33,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:33,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:33,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:33,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on 2e08a42276d589261ea162f9850b0f0a 2023-03-31 16:55:33,481 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2e08a42276d589261ea162f9850b0f0a 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-31 16:55:33,481 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e08a42276d589261ea162f9850b0f0a, store=A 2023-03-31 16:55:33,481 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:33,481 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e08a42276d589261ea162f9850b0f0a, store=B 2023-03-31 16:55:33,481 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:33,481 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e08a42276d589261ea162f9850b0f0a, store=C 2023-03-31 16:55:33,481 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:33,506 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:33,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 295 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53680 deadline: 1680281793499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:33,506 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:33,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53626 deadline: 1680281793500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:33,506 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:33,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53652 deadline: 1680281793501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:33,507 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:33,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53640 deadline: 1680281793502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:33,507 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:33,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 291 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53692 deadline: 1680281793504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:33,541 DEBUG [Thread-2550] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:33,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:33,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:33,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:33,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:33,544 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:33,544 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:33,544 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:33,544 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:33,544 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:33,544 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:33,545 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:33,545 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:33,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:33,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:33,546 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:33,546 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:33,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:33,546 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:33,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:33,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:33,546 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:33,547 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:33,547 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:33,547 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:33,547 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:33,547 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:33,547 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:33,547 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:33,547 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:33,547 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. started... 2023-03-31 16:55:33,547 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. as already flushing 2023-03-31 16:55:33,547 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:33,547 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:33,547 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:33,547 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:33,547 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:33,548 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:33,550 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:33,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:33,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:33,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:33,550 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:33,550 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:33,550 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:33,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:33,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:33,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:33,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:33,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:33,552 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:33,552 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28a43309[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:33,552 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:33,552 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:33,553 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:33,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:33,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:33,553 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:33,553 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:33,553 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:33,553 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:33,553 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:33,554 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:33,554 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:33,554 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:33,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:33,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:33,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:33,555 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:33,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:33,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:33,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:33,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:33,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:33,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:33,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:33,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:33,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:33,557 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:33,557 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:33,557 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:33,557 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:33,558 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:33,559 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:33,559 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:33,559 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:33,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:33,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:33,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:33,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:33,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:33,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:33,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:33,560 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:33,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:33,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:33,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:33,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:33,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:33,562 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:33,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:33,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:33,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:33,566 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:33,566 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:33,566 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:33,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:33,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:33,566 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:33,566 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:33,566 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:33,566 DEBUG [Thread-2550] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:33,566 DEBUG [Thread-2550] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:33,566 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:33,566 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:33,566 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:33,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:33,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:33,566 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:33,567 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:33,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:33,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:33,610 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:33,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53626 deadline: 1680281793607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:33,611 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:33,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 297 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53680 deadline: 1680281793607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:33,611 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:33,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 279 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53652 deadline: 1680281793607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:33,612 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:33,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53640 deadline: 1680281793608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:33,613 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:33,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 293 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53692 deadline: 1680281793609, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:33,666 DEBUG [Thread-2550] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:33,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:33,668 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:33,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:33,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:33,688 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:33,688 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:33,688 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:33,688 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:33,689 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:33,689 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:33,690 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:33,690 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:33,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:33,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:33,690 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:33,690 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:33,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:33,691 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:33,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:33,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:33,691 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:33,691 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:33,691 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:33,692 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:33,692 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:33,692 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:33,692 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:33,692 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:33,692 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:33,692 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. started... 2023-03-31 16:55:33,692 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. as already flushing 2023-03-31 16:55:33,692 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:33,692 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:33,692 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:33,692 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:33,692 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:33,692 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:33,694 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:33,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:33,694 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:33,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:33,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:33,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:33,695 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:33,695 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:33,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:33,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:33,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:33,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:33,696 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:33,696 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33d8ac77[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:33,696 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:33,696 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:33,698 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:33,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:33,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:33,698 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:33,698 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:33,698 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:33,698 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:33,698 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:33,700 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:33,701 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:33,701 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:33,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:33,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:33,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:33,701 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:33,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:33,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:33,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:33,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:33,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:33,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:33,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:33,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:33,703 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:33,703 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:33,703 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:33,703 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:33,703 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:33,703 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:33,705 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:33,705 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:33,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:33,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:33,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:33,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:33,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:33,705 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:33,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:33,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:33,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:33,705 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:33,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:33,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:33,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:33,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:33,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:33,706 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:33,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:33,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:33,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:33,709 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:33,709 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:33,709 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:33,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:33,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:33,709 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:33,709 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:33,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:33,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:33,709 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:33,710 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:33,710 DEBUG [Thread-2550] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:33,710 DEBUG [Thread-2550] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:33,709 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:33,710 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:33,710 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:33,710 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:33,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:33,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:33,810 DEBUG [Thread-2550] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:33,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:33,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:33,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:33,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:33,813 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:33,813 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:33,813 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:33,813 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:33,814 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:33,814 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:33,816 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:33,816 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:33,816 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:33,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:33,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:33,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53626 deadline: 1680281793811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:33,816 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:33,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 281 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53652 deadline: 1680281793812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:33,817 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:33,817 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:33,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:33,818 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:33,818 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:33,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 299 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53680 deadline: 1680281793812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:33,818 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:33,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:33,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:33,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 292 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53640 deadline: 1680281793813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:33,818 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:33,818 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:33,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 295 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53692 deadline: 1680281793814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:33,819 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:33,819 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:33,819 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:33,819 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:33,819 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:33,820 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:33,820 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:33,820 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:33,820 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. started... 2023-03-31 16:55:33,820 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. as already flushing 2023-03-31 16:55:33,820 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:33,820 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:33,820 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:33,820 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:33,820 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:33,820 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:33,822 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:33,822 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:33,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:33,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:33,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:33,822 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:33,822 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:33,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:33,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:33,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:33,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:33,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:33,824 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:33,824 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10cead31[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:33,824 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:33,824 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:33,825 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:33,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:33,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:33,826 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:33,826 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:33,826 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:33,826 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:33,826 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:33,827 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:33,827 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:33,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:33,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:33,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:33,827 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:33,828 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:33,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:33,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:33,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:33,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:33,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:33,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:33,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:33,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:33,830 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:33,830 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:33,830 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:33,830 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:33,830 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:33,830 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:33,833 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:33,834 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:33,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:33,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:33,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:33,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:33,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:33,834 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:33,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:33,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:33,834 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:33,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:33,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:33,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:33,835 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:33,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:33,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:33,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:33,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:33,841 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:33,841 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:33,841 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:33,841 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:33,841 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:33,841 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:33,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:33,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:33,841 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:33,842 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:33,842 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:33,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:33,842 DEBUG [Thread-2550] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:33,842 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:33,842 DEBUG [Thread-2550] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:33,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:33,842 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:33,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:33,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:33,893 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023033198806b6cc6f640d28923fecb96217854 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033198806b6cc6f640d28923fecb96217854 2023-03-31 16:55:33,900 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=974, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/A/94b5c60cfde04684901daf3ef54fca23 2023-03-31 16:55:33,910 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=974 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/B/ce5a4f549f8246c58e275e956e04a266 2023-03-31 16:55:33,919 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=974 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/C/1ee496d6c9e148c2aba748fe80a5e626 2023-03-31 16:55:33,923 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/A/94b5c60cfde04684901daf3ef54fca23 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/94b5c60cfde04684901daf3ef54fca23 2023-03-31 16:55:33,926 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/94b5c60cfde04684901daf3ef54fca23, entries=200, sequenceid=974, filesize=32.3 K 2023-03-31 16:55:33,926 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/B/ce5a4f549f8246c58e275e956e04a266 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/ce5a4f549f8246c58e275e956e04a266 2023-03-31 16:55:33,929 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/ce5a4f549f8246c58e275e956e04a266, entries=150, sequenceid=974, filesize=11.9 K 2023-03-31 16:55:33,930 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/C/1ee496d6c9e148c2aba748fe80a5e626 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/1ee496d6c9e148c2aba748fe80a5e626 2023-03-31 16:55:33,933 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/1ee496d6c9e148c2aba748fe80a5e626, entries=150, sequenceid=974, filesize=11.9 K 2023-03-31 16:55:33,933 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 2e08a42276d589261ea162f9850b0f0a in 452ms, sequenceid=974, compaction requested=true 2023-03-31 16:55:33,933 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2e08a42276d589261ea162f9850b0f0a: 2023-03-31 16:55:33,934 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:55:33,934 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 16:55:33,934 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 16:55:33,934 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:55:33,934 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:55:33,935 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85792 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 16:55:33,935 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37172 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 16:55:33,935 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1912): 2e08a42276d589261ea162f9850b0f0a/A is initiating minor compaction (all files) 2023-03-31 16:55:33,935 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): 2e08a42276d589261ea162f9850b0f0a/B is initiating minor compaction (all files) 2023-03-31 16:55:33,935 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2e08a42276d589261ea162f9850b0f0a/A in TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:33,935 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2e08a42276d589261ea162f9850b0f0a/B in TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:33,935 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/182e43fa1bfa4de78d67329ae8c08657, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/39e7cce7987c4200b462277008145ca9, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/94b5c60cfde04684901daf3ef54fca23] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp, totalSize=83.8 K 2023-03-31 16:55:33,935 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/dd693f1f178544038668b80e543944ae, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/234d0641c0ac4faf90398dcb73301abd, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/ce5a4f549f8246c58e275e956e04a266] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp, totalSize=36.3 K 2023-03-31 16:55:33,935 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 182e43fa1bfa4de78d67329ae8c08657, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=923, earliestPutTs=1680281731594 2023-03-31 16:55:33,935 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting dd693f1f178544038668b80e543944ae, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=923, earliestPutTs=1680281731594 2023-03-31 16:55:33,935 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 39e7cce7987c4200b462277008145ca9, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=959, earliestPutTs=1680281731708 2023-03-31 16:55:33,936 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 94b5c60cfde04684901daf3ef54fca23, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=974, earliestPutTs=1680281732854 2023-03-31 16:55:33,936 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 234d0641c0ac4faf90398dcb73301abd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=959, earliestPutTs=1680281731708 2023-03-31 16:55:33,936 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting ce5a4f549f8246c58e275e956e04a266, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=974, earliestPutTs=1680281732854 2023-03-31 16:55:33,942 DEBUG [Thread-2550] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:33,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:33,943 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): 2e08a42276d589261ea162f9850b0f0a#B#compaction#405 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:55:33,945 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:33,947 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2e08a42276d589261ea162f9850b0f0a#A#compaction#404 average throughput is 3.29 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:55:33,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:33,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:33,954 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:33,954 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:33,954 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:33,954 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:33,954 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:33,955 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:33,956 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:33,956 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:33,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:33,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:33,956 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:33,956 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:33,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:33,957 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:33,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:33,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:33,957 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:33,960 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:33,960 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:33,960 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:33,960 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:33,960 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:33,964 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:33,964 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:33,964 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:33,964 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. started... 2023-03-31 16:55:33,964 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2e08a42276d589261ea162f9850b0f0a 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-31 16:55:33,965 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e08a42276d589261ea162f9850b0f0a, store=A 2023-03-31 16:55:33,965 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:33,965 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e08a42276d589261ea162f9850b0f0a, store=B 2023-03-31 16:55:33,965 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:33,965 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/B/b47d1bbdb7584182b4ae82e6388a3968 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/b47d1bbdb7584182b4ae82e6388a3968 2023-03-31 16:55:33,965 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e08a42276d589261ea162f9850b0f0a, store=C 2023-03-31 16:55:33,965 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:33,969 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2e08a42276d589261ea162f9850b0f0a/B of 2e08a42276d589261ea162f9850b0f0a into b47d1bbdb7584182b4ae82e6388a3968(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:55:33,969 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2e08a42276d589261ea162f9850b0f0a: 2023-03-31 16:55:33,969 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a., storeName=2e08a42276d589261ea162f9850b0f0a/B, priority=13, startTime=1680281733934; duration=0sec 2023-03-31 16:55:33,970 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:55:33,970 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 16:55:33,972 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37172 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 16:55:33,972 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): 2e08a42276d589261ea162f9850b0f0a/C is initiating minor compaction (all files) 2023-03-31 16:55:33,972 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2e08a42276d589261ea162f9850b0f0a/C in TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:33,972 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/4aebc16902d54bfa8cdbd41aaf534fac, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/e94917e437c14c8abd6fa91fcdcf3a08, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/1ee496d6c9e148c2aba748fe80a5e626] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp, totalSize=36.3 K 2023-03-31 16:55:33,973 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 4aebc16902d54bfa8cdbd41aaf534fac, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=923, earliestPutTs=1680281731594 2023-03-31 16:55:33,973 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting e94917e437c14c8abd6fa91fcdcf3a08, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=959, earliestPutTs=1680281731708 2023-03-31 16:55:33,974 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 1ee496d6c9e148c2aba748fe80a5e626, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=974, earliestPutTs=1680281732854 2023-03-31 16:55:33,985 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): 2e08a42276d589261ea162f9850b0f0a#C#compaction#407 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:55:33,996 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023033190f03b3e253943eda8561ee7087531a4 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033190f03b3e253943eda8561ee7087531a4 2023-03-31 16:55:34,004 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/A/3887ac72e7b9430ebc8a1863f20a8b6b as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/3887ac72e7b9430ebc8a1863f20a8b6b 2023-03-31 16:55:34,009 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/C/709cc4443e6a4a10a2765e542fb5370e as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/709cc4443e6a4a10a2765e542fb5370e 2023-03-31 16:55:34,010 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2e08a42276d589261ea162f9850b0f0a/A of 2e08a42276d589261ea162f9850b0f0a into 3887ac72e7b9430ebc8a1863f20a8b6b(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:55:34,010 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2e08a42276d589261ea162f9850b0f0a: 2023-03-31 16:55:34,010 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a., storeName=2e08a42276d589261ea162f9850b0f0a/A, priority=13, startTime=1680281733933; duration=0sec 2023-03-31 16:55:34,010 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:55:34,014 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2e08a42276d589261ea162f9850b0f0a/C of 2e08a42276d589261ea162f9850b0f0a into 709cc4443e6a4a10a2765e542fb5370e(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:55:34,014 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2e08a42276d589261ea162f9850b0f0a: 2023-03-31 16:55:34,014 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a., storeName=2e08a42276d589261ea162f9850b0f0a/C, priority=13, startTime=1680281733934; duration=0sec 2023-03-31 16:55:34,014 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:55:34,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on 2e08a42276d589261ea162f9850b0f0a 2023-03-31 16:55:34,120 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. as already flushing 2023-03-31 16:55:34,134 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:34,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53626 deadline: 1680281794126, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:34,134 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:34,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 298 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53692 deadline: 1680281794127, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:34,135 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:34,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 285 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53652 deadline: 1680281794128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:34,135 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:34,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 303 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53680 deadline: 1680281794128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:34,136 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:34,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 296 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53640 deadline: 1680281794129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:34,237 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:34,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53626 deadline: 1680281794235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:34,237 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:34,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 300 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53692 deadline: 1680281794235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:34,238 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:34,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 287 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53652 deadline: 1680281794236, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:34,238 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:34,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 305 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53680 deadline: 1680281794236, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:34,238 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:34,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 298 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53640 deadline: 1680281794237, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:34,403 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=994, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/A/db22042b6cf240be9f0f9c79d436dfb3 2023-03-31 16:55:34,440 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:34,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53626 deadline: 1680281794438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:34,442 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:34,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 289 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53652 deadline: 1680281794439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:34,442 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:34,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 300 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53640 deadline: 1680281794439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:34,442 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:34,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 302 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53692 deadline: 1680281794439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:34,442 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:34,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 307 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53680 deadline: 1680281794440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:34,746 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:34,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 309 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53680 deadline: 1680281794743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:34,746 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:34,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53626 deadline: 1680281794743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:34,747 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:34,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 291 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53652 deadline: 1680281794744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:34,747 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:34,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 302 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53640 deadline: 1680281794745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:34,747 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:34,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 304 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53692 deadline: 1680281794745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:34,814 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=994 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/B/1615050641f44a78801f58e854d62fd8 2023-03-31 16:55:34,844 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=994 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/C/41fc20348b1145559664807f8b01e6de 2023-03-31 16:55:34,849 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/A/db22042b6cf240be9f0f9c79d436dfb3 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/db22042b6cf240be9f0f9c79d436dfb3 2023-03-31 16:55:34,853 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/db22042b6cf240be9f0f9c79d436dfb3, entries=150, sequenceid=994, filesize=25.4 K 2023-03-31 16:55:34,854 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/B/1615050641f44a78801f58e854d62fd8 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/1615050641f44a78801f58e854d62fd8 2023-03-31 16:55:34,858 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/1615050641f44a78801f58e854d62fd8, entries=150, sequenceid=994, filesize=11.9 K 2023-03-31 16:55:34,859 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/C/41fc20348b1145559664807f8b01e6de as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/41fc20348b1145559664807f8b01e6de 2023-03-31 16:55:34,863 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/41fc20348b1145559664807f8b01e6de, entries=150, sequenceid=994, filesize=11.9 K 2023-03-31 16:55:34,863 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 2e08a42276d589261ea162f9850b0f0a in 899ms, sequenceid=994, compaction requested=false 2023-03-31 16:55:34,864 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2e08a42276d589261ea162f9850b0f0a: 2023-03-31 16:55:34,864 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:34,864 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:34,864 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:34,864 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:34,864 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:34,864 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:34,866 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:34,866 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:34,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:34,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:34,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:34,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:34,866 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:34,866 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:34,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:34,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:34,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:34,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:34,867 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:34,867 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@260ba996[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:34,867 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:34,867 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:34,869 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:34,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:34,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:34,869 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:34,869 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:34,870 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:34,870 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:34,870 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:34,871 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:34,871 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:34,871 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:34,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:34,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:34,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:34,871 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:34,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:34,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:34,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:34,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:34,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:34,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:34,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:34,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:34,875 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:34,875 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:34,875 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:34,875 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:34,875 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:34,875 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:34,877 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:34,877 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:34,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:34,877 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:34,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:34,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:34,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:34,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:34,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:34,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:34,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:34,878 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:34,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:34,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:34,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:34,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:34,879 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:34,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:34,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:34,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:34,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:34,883 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:34,883 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:34,883 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:34,883 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:34,883 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:34,883 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:34,884 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:34,884 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:34,884 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:34,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:34,884 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:34,884 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:34,884 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:34,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:34,884 DEBUG [Thread-2550] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:34,884 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:34,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:34,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:34,884 DEBUG [Thread-2550] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:34,984 DEBUG [Thread-2550] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:34,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:34,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:34,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:34,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:34,988 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:34,988 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:34,988 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:34,988 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:34,989 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:34,989 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:34,991 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:34,992 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:34,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:34,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:34,992 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:34,992 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:34,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:34,993 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:34,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:34,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:34,993 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:34,993 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:34,993 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:34,993 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:34,993 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:34,993 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:34,994 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:34,994 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:34,994 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:34,994 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. started... 2023-03-31 16:55:34,994 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2e08a42276d589261ea162f9850b0f0a 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-31 16:55:34,994 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e08a42276d589261ea162f9850b0f0a, store=A 2023-03-31 16:55:34,994 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:34,994 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e08a42276d589261ea162f9850b0f0a, store=B 2023-03-31 16:55:34,994 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:34,994 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e08a42276d589261ea162f9850b0f0a, store=C 2023-03-31 16:55:34,994 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:35,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on 2e08a42276d589261ea162f9850b0f0a 2023-03-31 16:55:35,252 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. as already flushing 2023-03-31 16:55:35,278 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:35,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 309 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53692 deadline: 1680281795271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:35,278 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:35,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 306 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53640 deadline: 1680281795272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:35,280 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:35,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53626 deadline: 1680281795275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:35,280 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:35,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 315 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53680 deadline: 1680281795277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:35,281 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:35,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 297 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53652 deadline: 1680281795278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:35,386 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:35,387 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:35,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 311 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53692 deadline: 1680281795379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:35,388 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:35,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 308 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53640 deadline: 1680281795379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:35,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53626 deadline: 1680281795381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:35,388 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:35,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 317 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53680 deadline: 1680281795382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:35,389 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:35,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 299 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53652 deadline: 1680281795382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:35,409 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303317065d8bab8d94d23823f1e8308881227 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303317065d8bab8d94d23823f1e8308881227 2023-03-31 16:55:35,417 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1009, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/A/1050484c9d994e9d8e4410dc14b8c387 2023-03-31 16:55:35,428 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=1009 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/B/f82fbe0ffed44664978b11e2f2c50cc3 2023-03-31 16:55:35,596 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:35,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 313 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53692 deadline: 1680281795589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:35,596 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:35,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 301 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53652 deadline: 1680281795590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:35,596 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:35,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53626 deadline: 1680281795590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:35,597 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:35,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 319 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53680 deadline: 1680281795590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:35,597 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:35,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 310 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53640 deadline: 1680281795590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:35,866 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=1009 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/C/5801b0208de1445ab308aa7936612ee6 2023-03-31 16:55:35,870 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/A/1050484c9d994e9d8e4410dc14b8c387 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/1050484c9d994e9d8e4410dc14b8c387 2023-03-31 16:55:35,873 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/1050484c9d994e9d8e4410dc14b8c387, entries=150, sequenceid=1009, filesize=25.4 K 2023-03-31 16:55:35,874 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/B/f82fbe0ffed44664978b11e2f2c50cc3 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/f82fbe0ffed44664978b11e2f2c50cc3 2023-03-31 16:55:35,877 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/f82fbe0ffed44664978b11e2f2c50cc3, entries=150, sequenceid=1009, filesize=11.9 K 2023-03-31 16:55:35,877 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/C/5801b0208de1445ab308aa7936612ee6 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/5801b0208de1445ab308aa7936612ee6 2023-03-31 16:55:35,880 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/5801b0208de1445ab308aa7936612ee6, entries=150, sequenceid=1009, filesize=11.9 K 2023-03-31 16:55:35,881 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 2e08a42276d589261ea162f9850b0f0a in 887ms, sequenceid=1009, compaction requested=true 2023-03-31 16:55:35,881 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2e08a42276d589261ea162f9850b0f0a: 2023-03-31 16:55:35,881 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:35,881 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:35,881 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:35,881 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:35,881 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:35,881 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:35,884 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:35,884 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:35,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:35,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:35,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:35,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:35,884 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:35,884 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:35,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:35,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:35,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:35,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:35,885 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:35,885 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f3eea25[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:35,885 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:35,886 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:35,887 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:35,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:35,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:35,887 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:35,887 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:35,887 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:35,887 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:35,887 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:35,888 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:35,888 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:35,888 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:35,888 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:35,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:35,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:35,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:35,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:35,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:35,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:35,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:35,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:35,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:35,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:35,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:35,891 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:35,891 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:35,891 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:35,891 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:35,891 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:35,891 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:35,893 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:35,893 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:35,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:35,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:35,893 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:35,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:35,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:35,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:35,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:35,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:35,894 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:35,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:35,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:35,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:35,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:35,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:35,895 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:35,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:35,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:35,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:35,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:35,897 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:35,897 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:35,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:35,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:35,897 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:35,897 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:35,898 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:35,898 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:35,898 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:35,898 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:35,897 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:35,898 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:35,897 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:35,898 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:35,897 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:35,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:35,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:35,898 DEBUG [Thread-2550] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:35,898 DEBUG [Thread-2550] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:35,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on 2e08a42276d589261ea162f9850b0f0a 2023-03-31 16:55:35,902 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2e08a42276d589261ea162f9850b0f0a 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-03-31 16:55:35,902 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680281735269, max timestamp=1680281735901Num uniques 859; 2023-03-31 16:55:35,902 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 16:55:35,902 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680281735269, max timestamp=1680281735901Num uniques 859; 2023-03-31 16:55:35,902 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680281735269, max timestamp=1680281735901Num uniques 859; 2023-03-31 16:55:35,902 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 16:55:35,903 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 16:55:35,903 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e08a42276d589261ea162f9850b0f0a, store=A 2023-03-31 16:55:35,903 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:55:35,903 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e08a42276d589261ea162f9850b0f0a, store=B 2023-03-31 16:55:35,903 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:55:35,903 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e08a42276d589261ea162f9850b0f0a, store=C 2023-03-31 16:55:35,903 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:55:35,963 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680281735916, max timestamp=1680281735961Num uniques 859; 2023-03-31 16:55:35,963 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680281735916, max timestamp=1680281735961Num uniques 859; 2023-03-31 16:55:35,963 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680281735916, max timestamp=1680281735961Num uniques 859; 2023-03-31 16:55:35,963 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 16:55:35,963 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 16:55:35,963 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 16:55:35,995 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1680281735974, max timestamp=1680281735993Num uniques 1018; 2023-03-31 16:55:35,995 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=14558, suffix heap size=133840,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=318 2023-03-31 16:55:35,996 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1680281735974, max timestamp=1680281735993Num uniques 1018; 2023-03-31 16:55:35,996 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1680281735974, max timestamp=1680281735993Num uniques 1018; 2023-03-31 16:55:35,996 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=14558, suffix heap size=133840,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=318 2023-03-31 16:55:35,996 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=14558, suffix heap size=133840,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=318 2023-03-31 16:55:35,998 DEBUG [Thread-2550] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:35,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:36,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:36,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:36,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:36,001 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:36,001 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:36,001 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:36,002 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:36,002 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:36,002 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:36,003 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:36,003 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:36,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:36,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:36,004 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,004 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:36,004 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:36,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:36,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:36,004 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:36,005 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:36,005 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:36,005 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:36,005 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:36,005 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:36,005 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:36,005 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:36,006 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:36,006 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. started... 2023-03-31 16:55:36,006 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. as already flushing 2023-03-31 16:55:36,006 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:36,006 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:36,006 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:36,006 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,006 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:36,006 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:36,008 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:36,008 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:36,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:36,008 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:36,008 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:36,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:36,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:36,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:36,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:36,009 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:36,009 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@597340a1[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:36,009 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:36,009 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:36,012 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:36,012 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,012 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:36,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:36,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:36,012 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:36,012 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:36,012 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:36,013 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,013 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:36,013 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:36,013 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:36,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:36,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:36,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:36,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:36,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:36,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:36,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:36,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:36,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:36,016 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:36,016 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:36,016 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:36,016 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:36,016 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:36,018 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:36,018 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:36,018 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:36,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:36,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:36,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:36,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:36,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:36,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:36,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:36,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:36,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:36,019 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:36,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:36,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:36,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,021 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:36,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:36,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,028 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:36,028 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:36,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:36,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:36,028 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:36,028 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:36,028 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,028 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:36,028 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:36,028 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:36,029 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:36,029 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,029 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:36,029 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:36,029 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:36,029 DEBUG [Thread-2550] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:36,029 DEBUG [Thread-2550] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:36,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:36,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:36,039 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:36,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 325 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53692 deadline: 1680281796023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,040 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:36,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53626 deadline: 1680281796023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,040 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:36,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 314 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53652 deadline: 1680281796024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,040 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:36,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 332 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53680 deadline: 1680281796028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,041 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:36,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 322 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53640 deadline: 1680281796032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,129 DEBUG [Thread-2550] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:36,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:36,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:36,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:36,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:36,132 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:36,132 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:36,132 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:36,132 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:36,133 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:36,133 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:36,134 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:36,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:36,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:36,134 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:36,135 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,135 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:36,135 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:36,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:36,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:36,135 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:36,136 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:36,136 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:36,136 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:36,136 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:36,136 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:36,136 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:36,136 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:36,136 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:36,136 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. started... 2023-03-31 16:55:36,136 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. as already flushing 2023-03-31 16:55:36,136 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:36,137 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:36,137 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:36,137 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,137 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:36,137 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:36,139 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,139 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:36,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:36,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:36,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:36,139 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:36,139 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:36,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:36,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:36,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:36,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:36,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3cb1d99b[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:36,141 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:36,141 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:36,144 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:36,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:36,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:36,144 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:36,144 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:36,144 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:36,144 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,144 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:36,146 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,146 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:36,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:36,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:36,146 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:36,146 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:36,146 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:36,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:36,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 327 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53692 deadline: 1680281796140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,146 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:36,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53626 deadline: 1680281796141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,147 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:36,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:36,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 316 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53652 deadline: 1680281796141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,147 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:36,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 334 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53680 deadline: 1680281796141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:36,147 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:36,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 324 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53640 deadline: 1680281796142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:36,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:36,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:36,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:36,149 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:36,149 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:36,149 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:36,149 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:36,149 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:36,151 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:36,151 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:36,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:36,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:36,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:36,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:36,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:36,151 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:36,151 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:36,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:36,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:36,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:36,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:36,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:36,154 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:36,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:36,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:36,157 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,157 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:36,157 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:36,157 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:36,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:36,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:36,157 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:36,157 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:36,157 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:36,157 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:36,157 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,158 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:36,158 DEBUG [Thread-2550] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:36,158 DEBUG [Thread-2550] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:36,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:36,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:36,158 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:36,158 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:36,158 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:36,258 DEBUG [Thread-2550] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:36,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:36,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:36,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:36,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:36,261 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:36,261 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:36,261 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:36,261 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:36,261 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:36,261 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:36,264 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:36,264 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:36,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:36,264 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,264 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:36,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:36,265 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:36,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:36,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:36,265 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:36,265 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:36,265 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:36,265 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:36,265 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:36,266 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:36,266 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:36,266 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:36,266 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:36,266 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. started... 2023-03-31 16:55:36,266 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. as already flushing 2023-03-31 16:55:36,266 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:36,266 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:36,266 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:36,266 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,266 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:36,266 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:36,268 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:36,268 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:36,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:36,268 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:36,268 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:36,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:36,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:36,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:36,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:36,269 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:36,270 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@478d9082[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:36,270 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:36,270 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:36,271 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:36,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:36,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:36,271 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:36,271 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:36,271 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:36,271 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,271 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:36,273 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,273 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:36,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:36,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:36,273 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:36,273 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:36,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:36,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:36,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:36,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:36,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:36,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:36,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:36,275 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:36,275 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:36,275 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:36,275 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:36,275 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:36,278 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:36,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:36,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:36,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:36,278 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:36,278 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:36,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:36,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:36,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:36,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:36,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:36,279 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:36,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:36,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:36,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:36,279 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:36,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:36,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,282 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:36,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:36,282 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:36,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:36,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:36,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:36,282 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,283 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:36,283 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:36,283 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:36,283 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:36,283 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:36,283 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:36,283 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,283 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:36,283 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:36,283 DEBUG [Thread-2550] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:36,283 DEBUG [Thread-2550] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:36,283 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:36,316 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303318a93285b5a11431ca4acb229aa627eeb to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303318a93285b5a11431ca4acb229aa627eeb 2023-03-31 16:55:36,349 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:36,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 276 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53626 deadline: 1680281796347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,349 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:36,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 318 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53652 deadline: 1680281796348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,350 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:36,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 329 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53692 deadline: 1680281796348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,350 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:36,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 326 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53640 deadline: 1680281796348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,350 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:36,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 336 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53680 deadline: 1680281796349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,383 DEBUG [Thread-2550] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:36,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:36,385 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:36,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:36,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:36,386 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:36,386 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:36,386 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:36,386 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:36,387 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:36,387 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:36,388 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:36,388 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:36,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:36,388 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,388 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:36,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:36,388 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:36,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:36,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:36,389 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:36,389 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:36,389 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:36,389 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:36,389 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:36,389 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:36,389 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:36,390 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:36,390 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:36,390 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. started... 2023-03-31 16:55:36,390 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. as already flushing 2023-03-31 16:55:36,390 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:36,390 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:36,390 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:36,390 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,390 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:36,390 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:36,391 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,391 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:36,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:36,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:36,392 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:36,392 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:36,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:36,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:36,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:36,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:36,393 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:36,393 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f3a2594[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:36,393 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:36,393 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:36,395 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:36,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:36,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:36,395 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:36,395 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:36,395 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:36,395 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,395 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:36,397 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:36,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:36,397 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:36,397 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:36,397 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:36,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:36,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:36,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:36,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:36,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:36,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:36,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:36,399 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:36,399 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:36,399 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:36,399 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:36,399 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:36,400 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:36,400 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:36,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:36,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:36,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:36,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:36,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:36,400 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:36,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:36,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:36,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:36,401 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:36,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:36,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:36,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:36,402 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:36,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:36,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,405 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,405 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:36,405 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:36,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:36,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:36,405 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:36,406 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:36,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:36,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:36,405 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:36,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:36,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:36,406 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,406 DEBUG [Thread-2550] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:36,406 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:36,406 DEBUG [Thread-2550] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:36,406 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:36,406 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:36,506 DEBUG [Thread-2550] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:36,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:36,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:36,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:36,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:36,509 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:36,509 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:36,509 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:36,509 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:36,510 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:36,510 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:36,512 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,512 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:36,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:36,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:36,512 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,512 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:36,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:36,512 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:36,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:36,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:36,513 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:36,513 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:36,513 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:36,513 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:36,513 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:36,513 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:36,513 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:36,513 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:36,513 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:36,513 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. started... 2023-03-31 16:55:36,513 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. as already flushing 2023-03-31 16:55:36,514 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:36,514 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:36,514 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:36,514 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,514 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:36,514 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:36,519 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,519 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:36,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:36,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:36,519 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:36,519 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:36,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:36,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:36,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:36,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:36,520 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:36,520 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c0f189a[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:36,520 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:36,520 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:36,521 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:36,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:36,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:36,521 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:36,521 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:36,521 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,521 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:36,521 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:36,523 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:36,523 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,523 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:36,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:36,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:36,523 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:36,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:36,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:36,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:36,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:36,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:36,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:36,525 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:36,525 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:36,526 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:36,526 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:36,526 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:36,526 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:36,528 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:36,528 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:36,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:36,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:36,528 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:36,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:36,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:36,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:36,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:36,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:36,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:36,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:36,529 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:36,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:36,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:36,530 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:36,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:36,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,532 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,532 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:36,532 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:36,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:36,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:36,532 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:36,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:36,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:36,532 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:36,532 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:36,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:36,533 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,533 DEBUG [Thread-2550] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:36,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:36,533 DEBUG [Thread-2550] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:36,533 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:36,533 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:36,533 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:36,633 DEBUG [Thread-2550] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:36,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:36,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:36,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:36,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:36,636 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:36,636 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:36,636 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:36,636 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:36,636 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:36,636 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:36,637 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:36,637 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:36,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:36,638 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,638 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:36,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:36,638 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:36,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:36,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:36,638 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:36,638 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:36,639 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:36,639 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:36,639 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:36,639 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:36,639 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:36,639 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:36,639 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:36,639 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. started... 2023-03-31 16:55:36,639 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. as already flushing 2023-03-31 16:55:36,639 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:36,639 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:36,639 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:36,639 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,639 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:36,639 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:36,640 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:36,640 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:36,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:36,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:36,641 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:36,641 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:36,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:36,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:36,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:36,642 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:36,642 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e406f81[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:36,642 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:36,642 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:36,644 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:36,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:36,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:36,644 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:36,644 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:36,644 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:36,645 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,645 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:36,651 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:36,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:36,651 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:36,651 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:36,652 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:36,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:36,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:36,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:36,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:36,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:36,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:36,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:36,654 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:36,655 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:36,655 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:36,655 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:36,655 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:36,656 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:36,656 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:36,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:36,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:36,657 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:36,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:36,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:36,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:36,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:36,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:36,657 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:36,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 331 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53692 deadline: 1680281796650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,658 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:36,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 328 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53640 deadline: 1680281796652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,658 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:36,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 278 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53626 deadline: 1680281796652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,658 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:36,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 320 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53652 deadline: 1680281796652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,658 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:36,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 338 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53680 deadline: 1680281796653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:36,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:36,660 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:36,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:36,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:36,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,662 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:36,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:36,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,665 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:36,665 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:36,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:36,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:36,665 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,665 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:36,665 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:36,665 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:36,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:36,665 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,665 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:36,666 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:36,666 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:36,666 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:36,666 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:36,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:36,666 DEBUG [Thread-2550] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:36,666 DEBUG [Thread-2550] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:36,722 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1031, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/A/60d4884762e244e2b7f9847fce48b5f5 2023-03-31 16:55:36,742 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=1031 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/B/5ca670710c9f4ea3a7c21c1efef0273f 2023-03-31 16:55:36,766 DEBUG [Thread-2550] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:36,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:36,768 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:36,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:36,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:36,769 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:36,769 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:36,769 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:36,769 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:36,770 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:36,770 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:36,774 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:36,774 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:36,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:36,775 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:36,775 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:36,775 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:36,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:36,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:36,776 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:36,776 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:36,776 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:36,776 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:36,776 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:36,776 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:36,776 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:36,776 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:36,776 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:36,776 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. started... 2023-03-31 16:55:36,776 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. as already flushing 2023-03-31 16:55:36,776 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:36,776 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:36,777 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:36,777 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,777 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:36,777 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:36,778 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,778 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:36,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:36,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:36,779 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:36,779 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:36,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:36,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:36,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:36,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:36,780 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:36,780 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45e2eb4b[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:36,780 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:36,780 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:36,781 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:36,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:36,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:36,781 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:36,781 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:36,782 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:36,782 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,782 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:36,783 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,783 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:36,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:36,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:36,783 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:36,783 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:36,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:36,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:36,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:36,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:36,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:36,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:36,785 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:36,785 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:36,785 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:36,785 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:36,785 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:36,785 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:36,787 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:36,788 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:36,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:36,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:36,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:36,788 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:36,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:36,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:36,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:36,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:36,788 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:36,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:36,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:36,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:36,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:36,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,789 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:36,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:36,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,791 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,791 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:36,791 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:36,791 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:36,791 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:36,791 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:36,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:36,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:36,791 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:36,792 DEBUG [Thread-2550] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:36,792 DEBUG [Thread-2550] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:36,791 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:36,791 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:36,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:36,792 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:36,792 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:36,792 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:36,892 DEBUG [Thread-2550] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:36,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:36,894 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:36,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:36,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:36,895 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:36,895 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:36,895 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:36,895 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:36,896 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:36,896 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:36,897 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:36,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:36,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:36,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:36,898 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:36,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:36,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:36,898 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:36,900 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,900 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,900 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:36,908 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:36,908 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:36,909 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:36,909 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:36,909 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:36,909 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:36,909 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:36,909 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:36,909 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. started... 2023-03-31 16:55:36,909 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. as already flushing 2023-03-31 16:55:36,909 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:36,909 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:36,909 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:36,909 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,909 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:36,909 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:36,911 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,911 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:36,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:36,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:36,912 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:36,912 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:36,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:36,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:36,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:36,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:36,915 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:36,915 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b3a60e6[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:36,915 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:36,915 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:36,920 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:36,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:36,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:36,921 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:36,921 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:36,921 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,921 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:36,921 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:36,923 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:36,923 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:36,923 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:36,923 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:36,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:36,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:36,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:36,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:36,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:36,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:36,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:36,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:36,927 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:36,927 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:36,927 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:36,927 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:36,927 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:36,929 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:36,929 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:36,929 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:36,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:36,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:36,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:36,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:36,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:36,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:36,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:36,929 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:36,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:36,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:36,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:36,932 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:36,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:36,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:36,935 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:36,935 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,935 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:36,935 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:36,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:36,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:36,935 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:36,935 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:36,935 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:36,935 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:36,935 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:36,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:36,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:36,935 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:36,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:36,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:36,936 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:36,936 DEBUG [Thread-2550] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:36,936 DEBUG [Thread-2550] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:37,036 DEBUG [Thread-2550] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:37,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:37,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:37,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:37,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:37,040 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:37,040 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:37,040 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:37,040 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:37,040 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:37,040 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:37,042 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:37,042 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:37,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:37,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:37,042 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:37,042 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:37,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:37,043 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:37,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:37,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:37,043 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:37,043 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:37,044 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:37,044 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:37,044 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:37,044 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:37,044 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:37,044 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:37,044 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:37,044 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. started... 2023-03-31 16:55:37,044 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. as already flushing 2023-03-31 16:55:37,044 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:37,044 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:37,044 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:37,044 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:37,044 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:37,044 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:37,046 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:37,046 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:37,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:37,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:37,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:37,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:37,047 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:37,047 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:37,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:37,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:37,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:37,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:37,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:37,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2fd562d3[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:37,049 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:37,049 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:37,050 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:37,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:37,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:37,050 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:37,050 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:37,050 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:37,050 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:37,050 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:37,052 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:37,052 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:37,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:37,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:37,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:37,052 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:37,052 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:37,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:37,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:37,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:37,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:37,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:37,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:37,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:37,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:37,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:37,054 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:37,054 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:37,054 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:37,054 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:37,054 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:37,057 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:37,057 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:37,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:37,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:37,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:37,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:37,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:37,057 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:37,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:37,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:37,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:37,057 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:37,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:37,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:37,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:37,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:37,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:37,059 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:37,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:37,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:37,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:37,061 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:37,061 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:37,061 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:37,061 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:37,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:37,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:37,061 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:37,061 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:37,061 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:37,061 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:37,061 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:37,061 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:37,061 DEBUG [Thread-2550] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:37,061 DEBUG [Thread-2550] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:37,061 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:37,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:37,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:37,061 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:37,161 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:37,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 330 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53640 deadline: 1680281797160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:37,161 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:37,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 280 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53626 deadline: 1680281797160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:37,161 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:37,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 340 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53680 deadline: 1680281797160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:37,161 DEBUG [Thread-2550] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:37,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:37,163 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:37,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:37,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:37,164 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:37,164 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:37,164 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:37,164 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:37,165 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:37,165 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:37,166 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=1031 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/C/5e14f4b7546e42109d615738eaf78b9b 2023-03-31 16:55:37,166 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:37,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:37,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:37,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:37,167 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:37,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:37,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:37,168 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:37,168 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:37,168 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:37,168 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:37,168 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:37,168 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:37,168 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:37,168 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:37,168 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:37,168 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:37,168 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. started... 2023-03-31 16:55:37,168 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:37,168 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. as already flushing 2023-03-31 16:55:37,168 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:37,169 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:37,169 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:37,169 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:37,169 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:37,169 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:37,169 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:37,170 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:37,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 333 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53692 deadline: 1680281797162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:37,170 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:37,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:37,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:37,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:37,171 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:37,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:37,171 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:37,171 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:37,171 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/A/60d4884762e244e2b7f9847fce48b5f5 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/60d4884762e244e2b7f9847fce48b5f5 2023-03-31 16:55:37,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:37,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:37,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:37,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:37,172 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:37,173 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42843ccf[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:37,173 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:37,173 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:37,173 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:37,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 322 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53652 deadline: 1680281797162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:37,175 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:37,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:37,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:37,175 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/60d4884762e244e2b7f9847fce48b5f5, entries=250, sequenceid=1031, filesize=39.2 K 2023-03-31 16:55:37,175 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:37,175 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:37,175 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:37,175 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:37,175 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:37,176 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/B/5ca670710c9f4ea3a7c21c1efef0273f as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/5ca670710c9f4ea3a7c21c1efef0273f 2023-03-31 16:55:37,176 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:37,177 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:37,176 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:37,177 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:37,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:37,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:37,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:37,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:37,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:37,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:37,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:37,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:37,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:37,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:37,179 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/5ca670710c9f4ea3a7c21c1efef0273f, entries=150, sequenceid=1031, filesize=11.9 K 2023-03-31 16:55:37,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:37,179 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:37,179 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:37,179 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:37,179 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:37,179 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:37,179 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:37,180 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/C/5e14f4b7546e42109d615738eaf78b9b as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/5e14f4b7546e42109d615738eaf78b9b 2023-03-31 16:55:37,181 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:37,181 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:37,181 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:37,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:37,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:37,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:37,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:37,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:37,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:37,182 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:37,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:37,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:37,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:37,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:37,183 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:37,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:37,183 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/5e14f4b7546e42109d615738eaf78b9b, entries=150, sequenceid=1031, filesize=11.9 K 2023-03-31 16:55:37,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:37,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:37,184 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~86.11 KB/88176, currentSize=154.31 KB/158010 for 2e08a42276d589261ea162f9850b0f0a in 1282ms, sequenceid=1031, compaction requested=true 2023-03-31 16:55:37,184 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2e08a42276d589261ea162f9850b0f0a: 2023-03-31 16:55:37,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:37,184 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:55:37,184 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 16:55:37,184 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 16:55:37,184 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:55:37,184 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:55:37,185 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:37,185 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:37,185 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:37,185 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:37,185 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:37,185 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:37,185 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:37,185 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:37,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:37,185 DEBUG [Thread-2550] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:37,185 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:37,186 DEBUG [Thread-2550] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:37,186 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:37,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:37,186 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49460 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 16:55:37,185 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 118984 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 16:55:37,186 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): 2e08a42276d589261ea162f9850b0f0a/B is initiating minor compaction (all files) 2023-03-31 16:55:37,186 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1912): 2e08a42276d589261ea162f9850b0f0a/A is initiating minor compaction (all files) 2023-03-31 16:55:37,185 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:37,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:37,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:37,186 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2e08a42276d589261ea162f9850b0f0a/A in TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:37,186 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2e08a42276d589261ea162f9850b0f0a/B in TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:37,186 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:37,186 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/3887ac72e7b9430ebc8a1863f20a8b6b, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/db22042b6cf240be9f0f9c79d436dfb3, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/1050484c9d994e9d8e4410dc14b8c387, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/60d4884762e244e2b7f9847fce48b5f5] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp, totalSize=116.2 K 2023-03-31 16:55:37,186 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:37,186 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/b47d1bbdb7584182b4ae82e6388a3968, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/1615050641f44a78801f58e854d62fd8, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/f82fbe0ffed44664978b11e2f2c50cc3, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/5ca670710c9f4ea3a7c21c1efef0273f] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp, totalSize=48.3 K 2023-03-31 16:55:37,186 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 3887ac72e7b9430ebc8a1863f20a8b6b, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=974, earliestPutTs=1680281732854 2023-03-31 16:55:37,187 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting b47d1bbdb7584182b4ae82e6388a3968, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=974, earliestPutTs=1680281732854 2023-03-31 16:55:37,187 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting db22042b6cf240be9f0f9c79d436dfb3, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=994, earliestPutTs=1680281733498 2023-03-31 16:55:37,187 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 1615050641f44a78801f58e854d62fd8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=994, earliestPutTs=1680281733498 2023-03-31 16:55:37,187 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 1050484c9d994e9d8e4410dc14b8c387, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=1009, earliestPutTs=1680281734121 2023-03-31 16:55:37,187 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting f82fbe0ffed44664978b11e2f2c50cc3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1009, earliestPutTs=1680281734121 2023-03-31 16:55:37,187 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 60d4884762e244e2b7f9847fce48b5f5, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=1031, earliestPutTs=1680281735269 2023-03-31 16:55:37,187 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 5ca670710c9f4ea3a7c21c1efef0273f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1031, earliestPutTs=1680281735269 2023-03-31 16:55:37,193 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): 2e08a42276d589261ea162f9850b0f0a#B#compaction#417 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:55:37,195 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2e08a42276d589261ea162f9850b0f0a#A#compaction#416 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:55:37,286 DEBUG [Thread-2550] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:37,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:37,287 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:37,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:37,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:37,288 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:37,288 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:37,288 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:37,288 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:37,289 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:37,289 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:37,291 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:37,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:37,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:37,291 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:37,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:37,292 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:37,292 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:37,292 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:37,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:37,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:37,292 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:37,293 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:37,293 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:37,293 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:37,293 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:37,293 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:37,293 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:37,293 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:37,293 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:37,293 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. started... 2023-03-31 16:55:37,293 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2e08a42276d589261ea162f9850b0f0a 3/3 column families, dataSize=154.31 KB heapSize=429.84 KB 2023-03-31 16:55:37,293 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e08a42276d589261ea162f9850b0f0a, store=A 2023-03-31 16:55:37,294 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:55:37,294 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e08a42276d589261ea162f9850b0f0a, store=B 2023-03-31 16:55:37,294 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:55:37,294 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e08a42276d589261ea162f9850b0f0a, store=C 2023-03-31 16:55:37,294 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:55:37,604 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/B/489efefc3544480baec63bfba22d8dc1 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/489efefc3544480baec63bfba22d8dc1 2023-03-31 16:55:37,609 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2e08a42276d589261ea162f9850b0f0a/B of 2e08a42276d589261ea162f9850b0f0a into 489efefc3544480baec63bfba22d8dc1(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:55:37,609 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2e08a42276d589261ea162f9850b0f0a: 2023-03-31 16:55:37,609 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a., storeName=2e08a42276d589261ea162f9850b0f0a/B, priority=12, startTime=1680281737184; duration=0sec 2023-03-31 16:55:37,609 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:55:37,610 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 16:55:37,611 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49460 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 16:55:37,611 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): 2e08a42276d589261ea162f9850b0f0a/C is initiating minor compaction (all files) 2023-03-31 16:55:37,611 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2e08a42276d589261ea162f9850b0f0a/C in TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:37,611 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/709cc4443e6a4a10a2765e542fb5370e, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/41fc20348b1145559664807f8b01e6de, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/5801b0208de1445ab308aa7936612ee6, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/5e14f4b7546e42109d615738eaf78b9b] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp, totalSize=48.3 K 2023-03-31 16:55:37,612 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 709cc4443e6a4a10a2765e542fb5370e, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=974, earliestPutTs=1680281732854 2023-03-31 16:55:37,612 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 41fc20348b1145559664807f8b01e6de, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=994, earliestPutTs=1680281733498 2023-03-31 16:55:37,612 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 5801b0208de1445ab308aa7936612ee6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1009, earliestPutTs=1680281734121 2023-03-31 16:55:37,613 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 5e14f4b7546e42109d615738eaf78b9b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1031, earliestPutTs=1680281735269 2023-03-31 16:55:37,619 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): 2e08a42276d589261ea162f9850b0f0a#C#compaction#419 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:55:37,626 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/C/99b10d86ebec4d6aad6563e005d368bc as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/99b10d86ebec4d6aad6563e005d368bc 2023-03-31 16:55:37,629 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2e08a42276d589261ea162f9850b0f0a/C of 2e08a42276d589261ea162f9850b0f0a into 99b10d86ebec4d6aad6563e005d368bc(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:55:37,629 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2e08a42276d589261ea162f9850b0f0a: 2023-03-31 16:55:37,629 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a., storeName=2e08a42276d589261ea162f9850b0f0a/C, priority=12, startTime=1680281737184; duration=0sec 2023-03-31 16:55:37,629 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:55:37,707 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303310280fa79b7a44d45b833b56776dd32b0 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303310280fa79b7a44d45b833b56776dd32b0 2023-03-31 16:55:38,014 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/A/12bce22fbe89467caeb9d81b13e8d080 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/12bce22fbe89467caeb9d81b13e8d080 2023-03-31 16:55:38,022 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2e08a42276d589261ea162f9850b0f0a/A of 2e08a42276d589261ea162f9850b0f0a into 12bce22fbe89467caeb9d81b13e8d080(size=26.3 K), total size for store is 26.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:55:38,022 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2e08a42276d589261ea162f9850b0f0a: 2023-03-31 16:55:38,022 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a., storeName=2e08a42276d589261ea162f9850b0f0a/A, priority=12, startTime=1680281737184; duration=0sec 2023-03-31 16:55:38,022 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:55:38,112 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1085, memsize=51.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/A/7d5dca6039444ba68111c161aa0ac4a1 2023-03-31 16:55:38,121 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=1085 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/B/5d697182f6c34f16aff8a11888c54087 2023-03-31 16:55:38,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on 2e08a42276d589261ea162f9850b0f0a 2023-03-31 16:55:38,169 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. as already flushing 2023-03-31 16:55:38,177 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:38,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 344 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53680 deadline: 1680281798175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:38,180 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:38,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 333 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53640 deadline: 1680281798176, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:38,180 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:38,181 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:38,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 283 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53626 deadline: 1680281798176, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:38,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 335 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53692 deadline: 1680281798176, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:38,181 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:38,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 325 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53652 deadline: 1680281798177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:38,282 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:38,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 346 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53680 deadline: 1680281798278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:38,283 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:38,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 335 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53640 deadline: 1680281798281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:38,283 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:38,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 285 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53626 deadline: 1680281798282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:38,286 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:38,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 327 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53652 deadline: 1680281798282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:38,424 DEBUG [Thread-2553] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x10377b50 to 127.0.0.1:51916 2023-03-31 16:55:38,424 DEBUG [Thread-2553] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 16:55:38,427 DEBUG [Thread-2557] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6a35a28d to 127.0.0.1:51916 2023-03-31 16:55:38,427 DEBUG [Thread-2557] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 16:55:38,429 DEBUG [Thread-2555] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4366fcdd to 127.0.0.1:51916 2023-03-31 16:55:38,429 DEBUG [Thread-2555] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 16:55:38,431 DEBUG [Thread-2551] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4fe1db6a to 127.0.0.1:51916 2023-03-31 16:55:38,431 DEBUG [Thread-2551] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 16:55:38,433 DEBUG [Thread-2559] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5a1d4246 to 127.0.0.1:51916 2023-03-31 16:55:38,433 DEBUG [Thread-2559] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 16:55:38,484 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:38,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 337 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53640 deadline: 1680281798484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:38,485 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:38,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 348 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53680 deadline: 1680281798484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:38,485 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:38,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 287 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53626 deadline: 1680281798485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:38,488 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:38,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 329 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53652 deadline: 1680281798488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:38,535 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=1085 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/C/ecb6d6a47c624c2aa3b92728f627ba76 2023-03-31 16:55:38,538 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/A/7d5dca6039444ba68111c161aa0ac4a1 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/7d5dca6039444ba68111c161aa0ac4a1 2023-03-31 16:55:38,540 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/7d5dca6039444ba68111c161aa0ac4a1, entries=150, sequenceid=1085, filesize=25.4 K 2023-03-31 16:55:38,541 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/B/5d697182f6c34f16aff8a11888c54087 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/5d697182f6c34f16aff8a11888c54087 2023-03-31 16:55:38,544 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/5d697182f6c34f16aff8a11888c54087, entries=150, sequenceid=1085, filesize=11.9 K 2023-03-31 16:55:38,544 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/C/ecb6d6a47c624c2aa3b92728f627ba76 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/ecb6d6a47c624c2aa3b92728f627ba76 2023-03-31 16:55:38,547 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/ecb6d6a47c624c2aa3b92728f627ba76, entries=150, sequenceid=1085, filesize=11.9 K 2023-03-31 16:55:38,547 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~429.80 KB/440112, currentSize=33.54 KB/34350 for 2e08a42276d589261ea162f9850b0f0a in 1254ms, sequenceid=1085, compaction requested=false 2023-03-31 16:55:38,547 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2e08a42276d589261ea162f9850b0f0a: 2023-03-31 16:55:38,547 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:38,547 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:38,547 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:38,547 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:38,548 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:38,548 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:38,549 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:38,549 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:38,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:38,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:38,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:38,549 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:38,549 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:38,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:38,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:38,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:38,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:38,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:38,550 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:38,550 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10ee3f67[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:38,550 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:38,550 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:38,555 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:38,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:38,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:38,556 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:38,556 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:38,556 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:38,556 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:38,556 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:38,557 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:38,557 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:38,557 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:38,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:38,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:38,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:38,557 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:38,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:38,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:38,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:38,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:38,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:38,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:38,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:38,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:38,559 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:38,559 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:38,559 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:38,559 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:38,559 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:38,559 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:38,560 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:38,560 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:38,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:38,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:38,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:38,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:38,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:38,561 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:38,561 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:38,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:38,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:38,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:38,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:38,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:38,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:38,562 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:38,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:38,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:38,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:38,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:38,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:38,564 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:38,564 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:38,564 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:38,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:38,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:38,564 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:38,564 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:38,564 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:38,564 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:38,564 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:38,564 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:38,564 DEBUG [Thread-2550] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:38,564 DEBUG [Thread-2550] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:38,564 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:38,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:38,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:38,564 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:38,564 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:38,564 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:38,664 DEBUG [Thread-2550] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:38,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:38,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on 2e08a42276d589261ea162f9850b0f0a 2023-03-31 16:55:38,787 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2e08a42276d589261ea162f9850b0f0a 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-31 16:55:38,787 DEBUG [Thread-2542] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x49b5bd0d to 127.0.0.1:51916 2023-03-31 16:55:38,787 DEBUG [Thread-2548] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2c6cac96 to 127.0.0.1:51916 2023-03-31 16:55:38,787 DEBUG [Thread-2542] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 16:55:38,787 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e08a42276d589261ea162f9850b0f0a, store=A 2023-03-31 16:55:38,787 DEBUG [Thread-2548] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 16:55:38,787 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:38,787 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e08a42276d589261ea162f9850b0f0a, store=B 2023-03-31 16:55:38,787 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:38,787 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e08a42276d589261ea162f9850b0f0a, store=C 2023-03-31 16:55:38,787 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:38,789 DEBUG [Thread-2540] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x41507d16 to 127.0.0.1:51916 2023-03-31 16:55:38,789 DEBUG [Thread-2540] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 16:55:38,791 DEBUG [Thread-2544] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5c8d927d to 127.0.0.1:51916 2023-03-31 16:55:38,791 DEBUG [Thread-2544] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 16:55:39,197 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230331daf6d85287894a459d29911727e235f2 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331daf6d85287894a459d29911727e235f2 2023-03-31 16:55:39,602 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1098, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/A/dfbdce317ff648198e771a0dadee8943 2023-03-31 16:55:40,010 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=1098 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/B/56d25c2e89ec4cd9b7c646965b201055 2023-03-31 16:55:40,189 DEBUG [Thread-2546] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x419b69bf to 127.0.0.1:51916 2023-03-31 16:55:40,189 DEBUG [Thread-2546] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 16:55:40,189 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-31 16:55:40,189 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(394): wrote 184 2023-03-31 16:55:40,189 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(394): wrote 204 2023-03-31 16:55:40,189 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(394): wrote 196 2023-03-31 16:55:40,189 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(394): wrote 206 2023-03-31 16:55:40,189 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(394): wrote 223 2023-03-31 16:55:40,189 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-31 16:55:40,189 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-31 16:55:40,189 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(402): scanned 1779 2023-03-31 16:55:40,190 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(403): verified 5335 rows 2023-03-31 16:55:40,190 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(402): scanned 1791 2023-03-31 16:55:40,190 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(403): verified 5359 rows 2023-03-31 16:55:40,190 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(402): scanned 1785 2023-03-31 16:55:40,190 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(403): verified 5354 rows 2023-03-31 16:55:40,190 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(402): scanned 1792 2023-03-31 16:55:40,190 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(403): verified 5375 rows 2023-03-31 16:55:40,190 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(402): scanned 1793 2023-03-31 16:55:40,190 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(403): verified 5375 rows 2023-03-31 16:55:40,190 INFO [Listener at localhost/44135] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-31 16:55:40,190 DEBUG [Listener at localhost/44135] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4473f45c to 127.0.0.1:51916 2023-03-31 16:55:40,190 DEBUG [Listener at localhost/44135] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 16:55:40,193 INFO [Listener at localhost/44135] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-31 16:55:40,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-03-31 16:55:40,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure2.ProcedureExecutor(1029): Stored pid=43, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-31 16:55:40,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-31 16:55:40,197 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680281740197"}]},"ts":"1680281740197"} 2023-03-31 16:55:40,198 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-31 16:55:40,200 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-31 16:55:40,200 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2e08a42276d589261ea162f9850b0f0a, UNASSIGN}] 2023-03-31 16:55:40,201 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2e08a42276d589261ea162f9850b0f0a, UNASSIGN 2023-03-31 16:55:40,202 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=2e08a42276d589261ea162f9850b0f0a, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:40,202 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680281740202"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680281740202"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680281740202"}]},"ts":"1680281740202"} 2023-03-31 16:55:40,203 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=44, state=RUNNABLE; CloseRegionProcedure 2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375}] 2023-03-31 16:55:40,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-31 16:55:40,354 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 2e08a42276d589261ea162f9850b0f0a 2023-03-31 16:55:40,354 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 2e08a42276d589261ea162f9850b0f0a, disabling compactions & flushes 2023-03-31 16:55:40,354 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:40,418 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=1098 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/C/c3a083f898b84f28b1097509f10e28e3 2023-03-31 16:55:40,421 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/A/dfbdce317ff648198e771a0dadee8943 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/dfbdce317ff648198e771a0dadee8943 2023-03-31 16:55:40,424 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/dfbdce317ff648198e771a0dadee8943, entries=150, sequenceid=1098, filesize=25.4 K 2023-03-31 16:55:40,424 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/B/56d25c2e89ec4cd9b7c646965b201055 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/56d25c2e89ec4cd9b7c646965b201055 2023-03-31 16:55:40,427 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/56d25c2e89ec4cd9b7c646965b201055, entries=150, sequenceid=1098, filesize=11.9 K 2023-03-31 16:55:40,428 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/C/c3a083f898b84f28b1097509f10e28e3 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/c3a083f898b84f28b1097509f10e28e3 2023-03-31 16:55:40,431 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/c3a083f898b84f28b1097509f10e28e3, entries=150, sequenceid=1098, filesize=11.9 K 2023-03-31 16:55:40,431 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=20.13 KB/20610 for 2e08a42276d589261ea162f9850b0f0a in 1644ms, sequenceid=1098, compaction requested=true 2023-03-31 16:55:40,431 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2e08a42276d589261ea162f9850b0f0a: 2023-03-31 16:55:40,431 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:40,431 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. because compaction request was cancelled 2023-03-31 16:55:40,431 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:40,431 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:55:40,432 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. after waiting 0 ms 2023-03-31 16:55:40,432 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. because compaction request was cancelled 2023-03-31 16:55:40,432 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:40,432 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:55:40,432 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 2e08a42276d589261ea162f9850b0f0a 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-03-31 16:55:40,432 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. because compaction request was cancelled 2023-03-31 16:55:40,432 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:55:40,432 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e08a42276d589261ea162f9850b0f0a, store=A 2023-03-31 16:55:40,432 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:40,432 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e08a42276d589261ea162f9850b0f0a, store=B 2023-03-31 16:55:40,432 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:40,432 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2e08a42276d589261ea162f9850b0f0a, store=C 2023-03-31 16:55:40,432 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:40,441 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023033131062f7bf2f648ddbfb647dd6b67a40b to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033131062f7bf2f648ddbfb647dd6b67a40b 2023-03-31 16:55:40,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-31 16:55:40,705 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-03-31 16:55:40,730 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/a6620a784fb843c0b5dac9f70fc7b72f, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/ef864b2a48ed400f9032185238ea5797, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/f08fa1a4164348969e5d8d6b70130ddc, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/7f66cf024b254b04850b29763a69b7fb, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/ff84e982bb2b45989a8687854273af5c, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/014d28d712f045e2b3f8956f412b1d7b, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/4a436b177a4a47f9a4b515aabc45e480, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/0f8be71144434694b159132741a14b7d, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/5b257cd71994401e8c54fc47ab635770, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/d58a357d639b475f8d5aa7804a1a3910, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/3639fada1ae14f40822eb6ac85aae320, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/bb2910439e1f4f66b0ba0a460ff16d83, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/7c9ee1cd241e4e96b730d24e834a6810, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/ebcc5156327c453b9aaca0effda9648c, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/a1e50485a85d41bcb57d2662b361bf2a, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/87b9ebcb1ce0494981e9bc97efe3854d, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/02b785a12c57488aa86cfbedc928c28f, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/94c85cba95c14b7dbedf2288e21d394a, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/182e43fa1bfa4de78d67329ae8c08657, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/39e7cce7987c4200b462277008145ca9, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/94b5c60cfde04684901daf3ef54fca23, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/3887ac72e7b9430ebc8a1863f20a8b6b, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/db22042b6cf240be9f0f9c79d436dfb3, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/1050484c9d994e9d8e4410dc14b8c387, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/60d4884762e244e2b7f9847fce48b5f5] to archive 2023-03-31 16:55:40,731 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-03-31 16:55:40,733 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/a6620a784fb843c0b5dac9f70fc7b72f to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/a6620a784fb843c0b5dac9f70fc7b72f 2023-03-31 16:55:40,733 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/ef864b2a48ed400f9032185238ea5797 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/ef864b2a48ed400f9032185238ea5797 2023-03-31 16:55:40,734 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/f08fa1a4164348969e5d8d6b70130ddc to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/f08fa1a4164348969e5d8d6b70130ddc 2023-03-31 16:55:40,735 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/7f66cf024b254b04850b29763a69b7fb to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/7f66cf024b254b04850b29763a69b7fb 2023-03-31 16:55:40,736 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/ff84e982bb2b45989a8687854273af5c to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/ff84e982bb2b45989a8687854273af5c 2023-03-31 16:55:40,737 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/014d28d712f045e2b3f8956f412b1d7b to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/014d28d712f045e2b3f8956f412b1d7b 2023-03-31 16:55:40,738 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/4a436b177a4a47f9a4b515aabc45e480 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/4a436b177a4a47f9a4b515aabc45e480 2023-03-31 16:55:40,739 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/0f8be71144434694b159132741a14b7d to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/0f8be71144434694b159132741a14b7d 2023-03-31 16:55:40,740 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/5b257cd71994401e8c54fc47ab635770 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/5b257cd71994401e8c54fc47ab635770 2023-03-31 16:55:40,741 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/d58a357d639b475f8d5aa7804a1a3910 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/d58a357d639b475f8d5aa7804a1a3910 2023-03-31 16:55:40,741 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/3639fada1ae14f40822eb6ac85aae320 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/3639fada1ae14f40822eb6ac85aae320 2023-03-31 16:55:40,742 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/bb2910439e1f4f66b0ba0a460ff16d83 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/bb2910439e1f4f66b0ba0a460ff16d83 2023-03-31 16:55:40,743 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/7c9ee1cd241e4e96b730d24e834a6810 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/7c9ee1cd241e4e96b730d24e834a6810 2023-03-31 16:55:40,744 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/ebcc5156327c453b9aaca0effda9648c to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/ebcc5156327c453b9aaca0effda9648c 2023-03-31 16:55:40,745 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/a1e50485a85d41bcb57d2662b361bf2a to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/a1e50485a85d41bcb57d2662b361bf2a 2023-03-31 16:55:40,746 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/87b9ebcb1ce0494981e9bc97efe3854d to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/87b9ebcb1ce0494981e9bc97efe3854d 2023-03-31 16:55:40,747 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/02b785a12c57488aa86cfbedc928c28f to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/02b785a12c57488aa86cfbedc928c28f 2023-03-31 16:55:40,747 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/94c85cba95c14b7dbedf2288e21d394a to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/94c85cba95c14b7dbedf2288e21d394a 2023-03-31 16:55:40,748 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/182e43fa1bfa4de78d67329ae8c08657 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/182e43fa1bfa4de78d67329ae8c08657 2023-03-31 16:55:40,749 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/39e7cce7987c4200b462277008145ca9 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/39e7cce7987c4200b462277008145ca9 2023-03-31 16:55:40,750 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/94b5c60cfde04684901daf3ef54fca23 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/94b5c60cfde04684901daf3ef54fca23 2023-03-31 16:55:40,751 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/3887ac72e7b9430ebc8a1863f20a8b6b to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/3887ac72e7b9430ebc8a1863f20a8b6b 2023-03-31 16:55:40,752 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/db22042b6cf240be9f0f9c79d436dfb3 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/db22042b6cf240be9f0f9c79d436dfb3 2023-03-31 16:55:40,753 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/1050484c9d994e9d8e4410dc14b8c387 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/1050484c9d994e9d8e4410dc14b8c387 2023-03-31 16:55:40,753 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/60d4884762e244e2b7f9847fce48b5f5 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/60d4884762e244e2b7f9847fce48b5f5 2023-03-31 16:55:40,758 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/3398757e201b4b808ebe5458fbdd4b08, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/a93a710b824c4cb58db67f7631336f84, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/e4dc33135f8b4d76b967d8da7b33277d, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/83918833daea4429ab630c0475189bc4, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/6748de9188374af4b3a1c48fc271be98, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/0e162064eea4434f93abd4ca80c7e0b0, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/cc40b230b4c44c57a759febd44a67cde, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/de2d5603083a45ab87b93250d2806f7a, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/7c152e43c2fc4113ab8c4bcd81cc8f0f, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/eaa2ce95526a4b3c862ac2ac1f47ce59, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/7c2bc6ac8ad54b18a0e5d3c6323bee7b, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/22b8364570004027919520746f1e95cb, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/59bb0cd23e4345d8b99eb3fb5b54a608, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/8768938ae0e2465a9ad32cc15c28b647, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/0647850e30f64b9489f501cf0d5b40bf, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/bc2831e8666242daa0b2190eaa0e6279, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/9120096c80b84934930ca22943118bf4, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/dd693f1f178544038668b80e543944ae, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/f662a530ec434af89b3591078d3b62d6, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/234d0641c0ac4faf90398dcb73301abd, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/b47d1bbdb7584182b4ae82e6388a3968, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/ce5a4f549f8246c58e275e956e04a266, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/1615050641f44a78801f58e854d62fd8, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/f82fbe0ffed44664978b11e2f2c50cc3, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/5ca670710c9f4ea3a7c21c1efef0273f] to archive 2023-03-31 16:55:40,758 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-03-31 16:55:40,759 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/3398757e201b4b808ebe5458fbdd4b08 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/3398757e201b4b808ebe5458fbdd4b08 2023-03-31 16:55:40,760 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/a93a710b824c4cb58db67f7631336f84 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/a93a710b824c4cb58db67f7631336f84 2023-03-31 16:55:40,761 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/e4dc33135f8b4d76b967d8da7b33277d to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/e4dc33135f8b4d76b967d8da7b33277d 2023-03-31 16:55:40,762 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/83918833daea4429ab630c0475189bc4 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/83918833daea4429ab630c0475189bc4 2023-03-31 16:55:40,763 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/6748de9188374af4b3a1c48fc271be98 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/6748de9188374af4b3a1c48fc271be98 2023-03-31 16:55:40,763 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/0e162064eea4434f93abd4ca80c7e0b0 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/0e162064eea4434f93abd4ca80c7e0b0 2023-03-31 16:55:40,764 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/cc40b230b4c44c57a759febd44a67cde to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/cc40b230b4c44c57a759febd44a67cde 2023-03-31 16:55:40,765 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/de2d5603083a45ab87b93250d2806f7a to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/de2d5603083a45ab87b93250d2806f7a 2023-03-31 16:55:40,766 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/7c152e43c2fc4113ab8c4bcd81cc8f0f to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/7c152e43c2fc4113ab8c4bcd81cc8f0f 2023-03-31 16:55:40,766 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/eaa2ce95526a4b3c862ac2ac1f47ce59 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/eaa2ce95526a4b3c862ac2ac1f47ce59 2023-03-31 16:55:40,767 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/7c2bc6ac8ad54b18a0e5d3c6323bee7b to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/7c2bc6ac8ad54b18a0e5d3c6323bee7b 2023-03-31 16:55:40,768 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/22b8364570004027919520746f1e95cb to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/22b8364570004027919520746f1e95cb 2023-03-31 16:55:40,769 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/59bb0cd23e4345d8b99eb3fb5b54a608 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/59bb0cd23e4345d8b99eb3fb5b54a608 2023-03-31 16:55:40,770 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/8768938ae0e2465a9ad32cc15c28b647 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/8768938ae0e2465a9ad32cc15c28b647 2023-03-31 16:55:40,770 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/0647850e30f64b9489f501cf0d5b40bf to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/0647850e30f64b9489f501cf0d5b40bf 2023-03-31 16:55:40,771 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/bc2831e8666242daa0b2190eaa0e6279 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/bc2831e8666242daa0b2190eaa0e6279 2023-03-31 16:55:40,772 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/9120096c80b84934930ca22943118bf4 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/9120096c80b84934930ca22943118bf4 2023-03-31 16:55:40,773 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/dd693f1f178544038668b80e543944ae to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/dd693f1f178544038668b80e543944ae 2023-03-31 16:55:40,774 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/f662a530ec434af89b3591078d3b62d6 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/f662a530ec434af89b3591078d3b62d6 2023-03-31 16:55:40,775 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/234d0641c0ac4faf90398dcb73301abd to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/234d0641c0ac4faf90398dcb73301abd 2023-03-31 16:55:40,776 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/b47d1bbdb7584182b4ae82e6388a3968 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/b47d1bbdb7584182b4ae82e6388a3968 2023-03-31 16:55:40,777 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/ce5a4f549f8246c58e275e956e04a266 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/ce5a4f549f8246c58e275e956e04a266 2023-03-31 16:55:40,778 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/1615050641f44a78801f58e854d62fd8 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/1615050641f44a78801f58e854d62fd8 2023-03-31 16:55:40,779 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/f82fbe0ffed44664978b11e2f2c50cc3 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/f82fbe0ffed44664978b11e2f2c50cc3 2023-03-31 16:55:40,779 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/5ca670710c9f4ea3a7c21c1efef0273f to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/5ca670710c9f4ea3a7c21c1efef0273f 2023-03-31 16:55:40,784 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/1f02a13835ae4007b20b6fdd7086e4bb, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/bd01f8213f5f4f6c85bf197afdd0b409, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/45e8aff6aa3d4ca2a6be13aab773ba71, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/3c31de27ce984fbb8e5d708b806f4f6b, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/03adfdca2a134225883dcc18bf792edd, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/52817ba110684de1b130fe2e302ac448, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/810230a31af8452dbbf98d27ce8beab1, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/7de3424bc92c45ea9a85199bc283475f, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/1d62fb3174034eb892d2e8d4db7768a8, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/80ca5efe366e43c3824b35b85432b77e, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/1ba89f6be7674b9e9211c2a1e0b4639c, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/66504eb74c96489ab3110b8d4381fed3, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/5a611fbd198345499fc3295729f872cd, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/ed4d02d4245e48bf9392e6059a997c7f, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/954b9564506b4f5b88858d38b9b8804e, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/bf4b5703a27849f7abbf04a28df9a49c, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/7dda3dc51f514d4bb1d2d3b00e0ea79a, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/4aebc16902d54bfa8cdbd41aaf534fac, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/73bac03200104ecaad413793455e2ad6, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/e94917e437c14c8abd6fa91fcdcf3a08, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/709cc4443e6a4a10a2765e542fb5370e, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/1ee496d6c9e148c2aba748fe80a5e626, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/41fc20348b1145559664807f8b01e6de, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/5801b0208de1445ab308aa7936612ee6, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/5e14f4b7546e42109d615738eaf78b9b] to archive 2023-03-31 16:55:40,784 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-03-31 16:55:40,785 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/1f02a13835ae4007b20b6fdd7086e4bb to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/1f02a13835ae4007b20b6fdd7086e4bb 2023-03-31 16:55:40,786 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/bd01f8213f5f4f6c85bf197afdd0b409 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/bd01f8213f5f4f6c85bf197afdd0b409 2023-03-31 16:55:40,787 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/45e8aff6aa3d4ca2a6be13aab773ba71 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/45e8aff6aa3d4ca2a6be13aab773ba71 2023-03-31 16:55:40,788 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/3c31de27ce984fbb8e5d708b806f4f6b to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/3c31de27ce984fbb8e5d708b806f4f6b 2023-03-31 16:55:40,789 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/03adfdca2a134225883dcc18bf792edd to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/03adfdca2a134225883dcc18bf792edd 2023-03-31 16:55:40,789 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/52817ba110684de1b130fe2e302ac448 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/52817ba110684de1b130fe2e302ac448 2023-03-31 16:55:40,790 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/810230a31af8452dbbf98d27ce8beab1 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/810230a31af8452dbbf98d27ce8beab1 2023-03-31 16:55:40,791 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/7de3424bc92c45ea9a85199bc283475f to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/7de3424bc92c45ea9a85199bc283475f 2023-03-31 16:55:40,792 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/1d62fb3174034eb892d2e8d4db7768a8 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/1d62fb3174034eb892d2e8d4db7768a8 2023-03-31 16:55:40,793 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/80ca5efe366e43c3824b35b85432b77e to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/80ca5efe366e43c3824b35b85432b77e 2023-03-31 16:55:40,793 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/1ba89f6be7674b9e9211c2a1e0b4639c to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/1ba89f6be7674b9e9211c2a1e0b4639c 2023-03-31 16:55:40,794 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/66504eb74c96489ab3110b8d4381fed3 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/66504eb74c96489ab3110b8d4381fed3 2023-03-31 16:55:40,795 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/5a611fbd198345499fc3295729f872cd to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/5a611fbd198345499fc3295729f872cd 2023-03-31 16:55:40,796 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/ed4d02d4245e48bf9392e6059a997c7f to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/ed4d02d4245e48bf9392e6059a997c7f 2023-03-31 16:55:40,796 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/954b9564506b4f5b88858d38b9b8804e to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/954b9564506b4f5b88858d38b9b8804e 2023-03-31 16:55:40,797 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/bf4b5703a27849f7abbf04a28df9a49c to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/bf4b5703a27849f7abbf04a28df9a49c 2023-03-31 16:55:40,798 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/7dda3dc51f514d4bb1d2d3b00e0ea79a to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/7dda3dc51f514d4bb1d2d3b00e0ea79a 2023-03-31 16:55:40,799 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/4aebc16902d54bfa8cdbd41aaf534fac to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/4aebc16902d54bfa8cdbd41aaf534fac 2023-03-31 16:55:40,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-31 16:55:40,799 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/73bac03200104ecaad413793455e2ad6 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/73bac03200104ecaad413793455e2ad6 2023-03-31 16:55:40,800 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/e94917e437c14c8abd6fa91fcdcf3a08 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/e94917e437c14c8abd6fa91fcdcf3a08 2023-03-31 16:55:40,801 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/709cc4443e6a4a10a2765e542fb5370e to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/709cc4443e6a4a10a2765e542fb5370e 2023-03-31 16:55:40,801 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/1ee496d6c9e148c2aba748fe80a5e626 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/1ee496d6c9e148c2aba748fe80a5e626 2023-03-31 16:55:40,802 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/41fc20348b1145559664807f8b01e6de to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/41fc20348b1145559664807f8b01e6de 2023-03-31 16:55:40,803 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/5801b0208de1445ab308aa7936612ee6 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/5801b0208de1445ab308aa7936612ee6 2023-03-31 16:55:40,803 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/5e14f4b7546e42109d615738eaf78b9b to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/5e14f4b7546e42109d615738eaf78b9b 2023-03-31 16:55:40,847 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1104, memsize=6.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/A/b4e0bc1deca144dbbd892a1c77f0183b 2023-03-31 16:55:41,256 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=1104 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/B/77421ed528c8438899f80ec2f9667e5e 2023-03-31 16:55:41,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-31 16:55:41,664 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=1104 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/C/f5d30441f3f7470a97a7cdec3b3272f3 2023-03-31 16:55:41,668 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/A/b4e0bc1deca144dbbd892a1c77f0183b as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/b4e0bc1deca144dbbd892a1c77f0183b 2023-03-31 16:55:41,671 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/b4e0bc1deca144dbbd892a1c77f0183b, entries=150, sequenceid=1104, filesize=25.4 K 2023-03-31 16:55:41,672 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/B/77421ed528c8438899f80ec2f9667e5e as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/77421ed528c8438899f80ec2f9667e5e 2023-03-31 16:55:41,674 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/77421ed528c8438899f80ec2f9667e5e, entries=150, sequenceid=1104, filesize=11.9 K 2023-03-31 16:55:41,675 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/.tmp/C/f5d30441f3f7470a97a7cdec3b3272f3 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/f5d30441f3f7470a97a7cdec3b3272f3 2023-03-31 16:55:41,677 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/f5d30441f3f7470a97a7cdec3b3272f3, entries=150, sequenceid=1104, filesize=11.9 K 2023-03-31 16:55:41,678 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=0 B/0 for 2e08a42276d589261ea162f9850b0f0a in 1246ms, sequenceid=1104, compaction requested=true 2023-03-31 16:55:41,685 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/recovered.edits/1107.seqid, newMaxSeqId=1107, maxSeqId=4 2023-03-31 16:55:41,686 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a. 2023-03-31 16:55:41,686 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 2e08a42276d589261ea162f9850b0f0a: 2023-03-31 16:55:41,687 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 2e08a42276d589261ea162f9850b0f0a 2023-03-31 16:55:41,687 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=2e08a42276d589261ea162f9850b0f0a, regionState=CLOSED 2023-03-31 16:55:41,688 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680281741687"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680281741687"}]},"ts":"1680281741687"} 2023-03-31 16:55:41,689 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=44 2023-03-31 16:55:41,689 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=44, state=SUCCESS; CloseRegionProcedure 2e08a42276d589261ea162f9850b0f0a, server=jenkins-hbase4.apache.org,33441,1680281619375 in 1.4850 sec 2023-03-31 16:55:41,690 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=44, resume processing ppid=43 2023-03-31 16:55:41,690 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=44, ppid=43, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=2e08a42276d589261ea162f9850b0f0a, UNASSIGN in 1.4890 sec 2023-03-31 16:55:41,691 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680281741691"}]},"ts":"1680281741691"} 2023-03-31 16:55:41,691 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-31 16:55:41,693 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-31 16:55:41,694 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=43, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.4990 sec 2023-03-31 16:55:42,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-31 16:55:42,301 INFO [Listener at localhost/44135] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 43 completed 2023-03-31 16:55:42,302 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-03-31 16:55:42,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure2.ProcedureExecutor(1029): Stored pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-31 16:55:42,304 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-31 16:55:42,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-03-31 16:55:42,304 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=46, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-31 16:55:42,308 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a 2023-03-31 16:55:42,309 DEBUG [HFileArchiver-9] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A, FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B, FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C, FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/recovered.edits] 2023-03-31 16:55:42,311 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/12bce22fbe89467caeb9d81b13e8d080 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/12bce22fbe89467caeb9d81b13e8d080 2023-03-31 16:55:42,312 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/7d5dca6039444ba68111c161aa0ac4a1 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/7d5dca6039444ba68111c161aa0ac4a1 2023-03-31 16:55:42,313 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/b4e0bc1deca144dbbd892a1c77f0183b to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/b4e0bc1deca144dbbd892a1c77f0183b 2023-03-31 16:55:42,314 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/dfbdce317ff648198e771a0dadee8943 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/A/dfbdce317ff648198e771a0dadee8943 2023-03-31 16:55:42,316 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/489efefc3544480baec63bfba22d8dc1 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/489efefc3544480baec63bfba22d8dc1 2023-03-31 16:55:42,317 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/56d25c2e89ec4cd9b7c646965b201055 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/56d25c2e89ec4cd9b7c646965b201055 2023-03-31 16:55:42,318 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/5d697182f6c34f16aff8a11888c54087 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/5d697182f6c34f16aff8a11888c54087 2023-03-31 16:55:42,319 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/77421ed528c8438899f80ec2f9667e5e to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/B/77421ed528c8438899f80ec2f9667e5e 2023-03-31 16:55:42,321 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/99b10d86ebec4d6aad6563e005d368bc to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/99b10d86ebec4d6aad6563e005d368bc 2023-03-31 16:55:42,325 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/c3a083f898b84f28b1097509f10e28e3 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/c3a083f898b84f28b1097509f10e28e3 2023-03-31 16:55:42,326 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/ecb6d6a47c624c2aa3b92728f627ba76 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/ecb6d6a47c624c2aa3b92728f627ba76 2023-03-31 16:55:42,327 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/f5d30441f3f7470a97a7cdec3b3272f3 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/C/f5d30441f3f7470a97a7cdec3b3272f3 2023-03-31 16:55:42,329 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/recovered.edits/1107.seqid to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a/recovered.edits/1107.seqid 2023-03-31 16:55:42,329 DEBUG [HFileArchiver-9] backup.HFileArchiver(596): Deleted hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/2e08a42276d589261ea162f9850b0f0a 2023-03-31 16:55:42,329 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-31 16:55:42,329 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-31 16:55:42,330 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-03-31 16:55:42,332 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303310280fa79b7a44d45b833b56776dd32b0 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303310280fa79b7a44d45b833b56776dd32b0 2023-03-31 16:55:42,332 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331060015725ce24b15b742333512e6117a to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331060015725ce24b15b742333512e6117a 2023-03-31 16:55:42,333 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303310baf7e1beb1f4b398ba738a595dbb7bd to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303310baf7e1beb1f4b398ba738a595dbb7bd 2023-03-31 16:55:42,334 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303310ce44b7f778746ae8bbadc0ed2d53991 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303310ce44b7f778746ae8bbadc0ed2d53991 2023-03-31 16:55:42,335 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033112fa8487d63948f2840a8996a0f27ec6 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033112fa8487d63948f2840a8996a0f27ec6 2023-03-31 16:55:42,336 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033118baebdd303243baaaa18dba03a2697d to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033118baebdd303243baaaa18dba03a2697d 2023-03-31 16:55:42,337 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303311b036c53697d4ac7838935b3978053f1 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303311b036c53697d4ac7838935b3978053f1 2023-03-31 16:55:42,337 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033131062f7bf2f648ddbfb647dd6b67a40b to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033131062f7bf2f648ddbfb647dd6b67a40b 2023-03-31 16:55:42,338 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033139dc441e46f04946b79eb9b9613ee48e to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033139dc441e46f04946b79eb9b9613ee48e 2023-03-31 16:55:42,339 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303313a14f41ddaf54a4e92ec9ed3b3ba7958 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303313a14f41ddaf54a4e92ec9ed3b3ba7958 2023-03-31 16:55:42,340 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033147da32ecffe74422808fd20597904b30 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033147da32ecffe74422808fd20597904b30 2023-03-31 16:55:42,341 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303315898bd95130848da8d496a432bdd19c8 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303315898bd95130848da8d496a432bdd19c8 2023-03-31 16:55:42,342 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303317065d8bab8d94d23823f1e8308881227 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303317065d8bab8d94d23823f1e8308881227 2023-03-31 16:55:42,342 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033172a1bf808f674bb8887cffa379a7e5dd to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033172a1bf808f674bb8887cffa379a7e5dd 2023-03-31 16:55:42,343 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303318a93285b5a11431ca4acb229aa627eeb to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303318a93285b5a11431ca4acb229aa627eeb 2023-03-31 16:55:42,344 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033190f03b3e253943eda8561ee7087531a4 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033190f03b3e253943eda8561ee7087531a4 2023-03-31 16:55:42,345 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033198806b6cc6f640d28923fecb96217854 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033198806b6cc6f640d28923fecb96217854 2023-03-31 16:55:42,345 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331b1386d3f7dc441eb88a1e4ccb4063787 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331b1386d3f7dc441eb88a1e4ccb4063787 2023-03-31 16:55:42,346 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331c801fe5bc51943c1881da19c323f5b75 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331c801fe5bc51943c1881da19c323f5b75 2023-03-31 16:55:42,347 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331daf6d85287894a459d29911727e235f2 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331daf6d85287894a459d29911727e235f2 2023-03-31 16:55:42,348 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331ee96385ef7c04befbce6add257fe7f5e to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331ee96385ef7c04befbce6add257fe7f5e 2023-03-31 16:55:42,349 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331f81243e16ce6403bb3fb39933b7feb23 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331f81243e16ce6403bb3fb39933b7feb23 2023-03-31 16:55:42,349 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-31 16:55:42,351 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=46, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-31 16:55:42,352 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-31 16:55:42,353 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-31 16:55:42,354 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=46, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-31 16:55:42,354 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-31 16:55:42,354 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680281742354"}]},"ts":"9223372036854775807"} 2023-03-31 16:55:42,355 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-31 16:55:42,355 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 2e08a42276d589261ea162f9850b0f0a, NAME => 'TestAcidGuarantees,,1680281716355.2e08a42276d589261ea162f9850b0f0a.', STARTKEY => '', ENDKEY => ''}] 2023-03-31 16:55:42,355 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-31 16:55:42,355 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680281742355"}]},"ts":"9223372036854775807"} 2023-03-31 16:55:42,356 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-31 16:55:42,358 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=46, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-31 16:55:42,358 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=46, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 56 msec 2023-03-31 16:55:42,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-03-31 16:55:42,405 INFO [Listener at localhost/44135] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 46 completed 2023-03-31 16:55:42,420 INFO [Listener at localhost/44135] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=237 (was 224) - Thread LEAK? -, OpenFileDescriptor=560 (was 583), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=384 (was 390), ProcessCount=169 (was 169), AvailableMemoryMB=3606 (was 3538) - AvailableMemoryMB LEAK? - 2023-03-31 16:55:42,436 INFO [Listener at localhost/44135] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=237, OpenFileDescriptor=560, MaxFileDescriptor=60000, SystemLoadAverage=384, ProcessCount=169, AvailableMemoryMB=3605 2023-03-31 16:55:42,438 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-31 16:55:42,439 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-31 16:55:42,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure2.ProcedureExecutor(1029): Stored pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-31 16:55:42,441 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-31 16:55:42,442 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 47 2023-03-31 16:55:42,442 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-31 16:55:42,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-31 16:55:42,443 DEBUG [HFileArchiver-10] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5 2023-03-31 16:55:42,444 DEBUG [HFileArchiver-10] backup.HFileArchiver(153): Directory hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5 empty. 2023-03-31 16:55:42,444 DEBUG [HFileArchiver-10] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5 2023-03-31 16:55:42,444 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-31 16:55:42,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-31 16:55:42,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-31 16:55:42,854 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-31 16:55:42,856 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => fd96548248d41c209e12fca5d7040af5, NAME => 'TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp 2023-03-31 16:55:42,868 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-31 16:55:42,869 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing fd96548248d41c209e12fca5d7040af5, disabling compactions & flushes 2023-03-31 16:55:42,869 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:42,869 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:42,869 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. after waiting 0 ms 2023-03-31 16:55:42,869 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:42,869 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:42,869 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for fd96548248d41c209e12fca5d7040af5: 2023-03-31 16:55:42,871 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-31 16:55:42,871 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680281742871"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680281742871"}]},"ts":"1680281742871"} 2023-03-31 16:55:42,872 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-31 16:55:42,873 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-31 16:55:42,873 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680281742873"}]},"ts":"1680281742873"} 2023-03-31 16:55:42,873 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-31 16:55:42,881 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=fd96548248d41c209e12fca5d7040af5, ASSIGN}] 2023-03-31 16:55:42,883 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=fd96548248d41c209e12fca5d7040af5, ASSIGN 2023-03-31 16:55:42,883 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=fd96548248d41c209e12fca5d7040af5, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,33441,1680281619375; forceNewPlan=false, retain=false 2023-03-31 16:55:43,034 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=fd96548248d41c209e12fca5d7040af5, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:43,035 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680281743034"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680281743034"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680281743034"}]},"ts":"1680281743034"} 2023-03-31 16:55:43,036 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=49, ppid=48, state=RUNNABLE; OpenRegionProcedure fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375}] 2023-03-31 16:55:43,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-31 16:55:43,189 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:43,189 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => fd96548248d41c209e12fca5d7040af5, NAME => 'TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.', STARTKEY => '', ENDKEY => ''} 2023-03-31 16:55:43,190 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees fd96548248d41c209e12fca5d7040af5 2023-03-31 16:55:43,190 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-31 16:55:43,190 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for fd96548248d41c209e12fca5d7040af5 2023-03-31 16:55:43,190 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for fd96548248d41c209e12fca5d7040af5 2023-03-31 16:55:43,191 INFO [StoreOpener-fd96548248d41c209e12fca5d7040af5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region fd96548248d41c209e12fca5d7040af5 2023-03-31 16:55:43,192 DEBUG [StoreOpener-fd96548248d41c209e12fca5d7040af5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A 2023-03-31 16:55:43,192 DEBUG [StoreOpener-fd96548248d41c209e12fca5d7040af5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A 2023-03-31 16:55:43,192 INFO [StoreOpener-fd96548248d41c209e12fca5d7040af5-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-31 16:55:43,192 INFO [StoreOpener-fd96548248d41c209e12fca5d7040af5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region fd96548248d41c209e12fca5d7040af5 columnFamilyName A 2023-03-31 16:55:43,193 INFO [StoreOpener-fd96548248d41c209e12fca5d7040af5-1] regionserver.HStore(310): Store=fd96548248d41c209e12fca5d7040af5/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-31 16:55:43,193 INFO [StoreOpener-fd96548248d41c209e12fca5d7040af5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region fd96548248d41c209e12fca5d7040af5 2023-03-31 16:55:43,194 DEBUG [StoreOpener-fd96548248d41c209e12fca5d7040af5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B 2023-03-31 16:55:43,194 DEBUG [StoreOpener-fd96548248d41c209e12fca5d7040af5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B 2023-03-31 16:55:43,194 INFO [StoreOpener-fd96548248d41c209e12fca5d7040af5-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-31 16:55:43,194 INFO [StoreOpener-fd96548248d41c209e12fca5d7040af5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region fd96548248d41c209e12fca5d7040af5 columnFamilyName B 2023-03-31 16:55:43,194 INFO [StoreOpener-fd96548248d41c209e12fca5d7040af5-1] regionserver.HStore(310): Store=fd96548248d41c209e12fca5d7040af5/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-31 16:55:43,195 INFO [StoreOpener-fd96548248d41c209e12fca5d7040af5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region fd96548248d41c209e12fca5d7040af5 2023-03-31 16:55:43,195 DEBUG [StoreOpener-fd96548248d41c209e12fca5d7040af5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C 2023-03-31 16:55:43,195 DEBUG [StoreOpener-fd96548248d41c209e12fca5d7040af5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C 2023-03-31 16:55:43,196 INFO [StoreOpener-fd96548248d41c209e12fca5d7040af5-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-31 16:55:43,196 INFO [StoreOpener-fd96548248d41c209e12fca5d7040af5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region fd96548248d41c209e12fca5d7040af5 columnFamilyName C 2023-03-31 16:55:43,196 INFO [StoreOpener-fd96548248d41c209e12fca5d7040af5-1] regionserver.HStore(310): Store=fd96548248d41c209e12fca5d7040af5/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-31 16:55:43,196 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:43,197 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5 2023-03-31 16:55:43,197 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5 2023-03-31 16:55:43,198 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-31 16:55:43,199 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for fd96548248d41c209e12fca5d7040af5 2023-03-31 16:55:43,200 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-31 16:55:43,200 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened fd96548248d41c209e12fca5d7040af5; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59641479, jitterRate=-0.11127270758152008}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-31 16:55:43,200 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for fd96548248d41c209e12fca5d7040af5: 2023-03-31 16:55:43,201 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5., pid=49, masterSystemTime=1680281743187 2023-03-31 16:55:43,202 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:43,202 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:43,202 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=fd96548248d41c209e12fca5d7040af5, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:43,202 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680281743202"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680281743202"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680281743202"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680281743202"}]},"ts":"1680281743202"} 2023-03-31 16:55:43,204 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=49, resume processing ppid=48 2023-03-31 16:55:43,204 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=49, ppid=48, state=SUCCESS; OpenRegionProcedure fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 in 167 msec 2023-03-31 16:55:43,205 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=47 2023-03-31 16:55:43,205 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=47, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=fd96548248d41c209e12fca5d7040af5, ASSIGN in 323 msec 2023-03-31 16:55:43,205 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-31 16:55:43,206 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680281743205"}]},"ts":"1680281743205"} 2023-03-31 16:55:43,206 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-31 16:55:43,208 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-31 16:55:43,209 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=47, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 769 msec 2023-03-31 16:55:43,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-31 16:55:43,546 INFO [Listener at localhost/44135] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 47 completed 2023-03-31 16:55:43,546 DEBUG [Listener at localhost/44135] zookeeper.ReadOnlyZKClient(139): Connect 0x20ed9117 to 127.0.0.1:51916 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 16:55:43,551 DEBUG [Listener at localhost/44135] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@34ad4aa8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 16:55:43,552 DEBUG [Listener at localhost/44135] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 16:55:43,554 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35484, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 16:55:43,555 DEBUG [Listener at localhost/44135] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-31 16:55:43,556 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55358, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-31 16:55:43,557 DEBUG [Listener at localhost/44135] zookeeper.ReadOnlyZKClient(139): Connect 0x50002343 to 127.0.0.1:51916 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 16:55:43,562 DEBUG [Listener at localhost/44135] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1c29757d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 16:55:43,562 DEBUG [Listener at localhost/44135] zookeeper.ReadOnlyZKClient(139): Connect 0x5726fb8b to 127.0.0.1:51916 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 16:55:43,566 DEBUG [Listener at localhost/44135] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6edd017, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 16:55:43,566 DEBUG [Listener at localhost/44135] zookeeper.ReadOnlyZKClient(139): Connect 0x6f2b3c03 to 127.0.0.1:51916 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 16:55:43,569 DEBUG [Listener at localhost/44135] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@fb168ce, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 16:55:43,569 DEBUG [Listener at localhost/44135] zookeeper.ReadOnlyZKClient(139): Connect 0x70e2be04 to 127.0.0.1:51916 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 16:55:43,573 DEBUG [Listener at localhost/44135] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@461f5f3f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 16:55:43,573 DEBUG [Listener at localhost/44135] zookeeper.ReadOnlyZKClient(139): Connect 0x067ea274 to 127.0.0.1:51916 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 16:55:43,578 DEBUG [Listener at localhost/44135] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@63093e21, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 16:55:43,578 DEBUG [Listener at localhost/44135] zookeeper.ReadOnlyZKClient(139): Connect 0x0b475d8d to 127.0.0.1:51916 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 16:55:43,581 DEBUG [Listener at localhost/44135] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@211582f2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 16:55:43,582 DEBUG [Listener at localhost/44135] zookeeper.ReadOnlyZKClient(139): Connect 0x549bef45 to 127.0.0.1:51916 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 16:55:43,585 DEBUG [Listener at localhost/44135] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@65816b6e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 16:55:43,585 DEBUG [Listener at localhost/44135] zookeeper.ReadOnlyZKClient(139): Connect 0x6ed183a4 to 127.0.0.1:51916 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 16:55:43,589 DEBUG [Listener at localhost/44135] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@653bb8fc, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 16:55:43,589 DEBUG [Listener at localhost/44135] zookeeper.ReadOnlyZKClient(139): Connect 0x10a344e6 to 127.0.0.1:51916 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 16:55:43,593 DEBUG [Listener at localhost/44135] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@720d29f5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 16:55:43,594 DEBUG [Listener at localhost/44135] zookeeper.ReadOnlyZKClient(139): Connect 0x3a9f3972 to 127.0.0.1:51916 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 16:55:43,597 DEBUG [Listener at localhost/44135] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3011deb3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 16:55:43,602 DEBUG [hconnection-0x66f2fb92-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 16:55:43,602 DEBUG [hconnection-0x7941f1d0-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 16:55:43,603 DEBUG [hconnection-0x4bbea1a5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 16:55:43,603 DEBUG [hconnection-0x2149da82-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 16:55:43,603 DEBUG [hconnection-0x590b0661-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 16:55:43,603 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35496, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 16:55:43,609 DEBUG [hconnection-0x5a60453-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 16:55:43,609 DEBUG [hconnection-0x51634792-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 16:55:43,609 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35538, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 16:55:43,611 DEBUG [hconnection-0x6c56a5fe-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 16:55:43,612 DEBUG [hconnection-0x47a958fd-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 16:55:43,609 DEBUG [hconnection-0x8684a7c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 16:55:43,612 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35516, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 16:55:43,614 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35494, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 16:55:43,616 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35522, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 16:55:43,616 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35564, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 16:55:43,617 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35558, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 16:55:43,621 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35546, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 16:55:43,622 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35580, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 16:55:43,624 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35582, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 16:55:43,625 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:43,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:43,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:43,631 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:43,631 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:43,631 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:43,631 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:43,631 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:43,631 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:43,632 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:43,632 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:43,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:43,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:43,633 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:43,633 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:43,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:43,633 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:43,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:43,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:43,633 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:43,634 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:43,634 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:43,634 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:43,634 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:43,634 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:43,634 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:43,634 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:43,634 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:43,634 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. started... 2023-03-31 16:55:43,635 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fd96548248d41c209e12fca5d7040af5 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-03-31 16:55:43,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on fd96548248d41c209e12fca5d7040af5 2023-03-31 16:55:43,654 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. as already flushing 2023-03-31 16:55:43,654 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=A 2023-03-31 16:55:43,654 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:43,655 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=B 2023-03-31 16:55:43,655 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:43,655 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=C 2023-03-31 16:55:43,655 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:43,670 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:43,670 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:43,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35582 deadline: 1680281803669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:43,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35580 deadline: 1680281803669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:43,671 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:43,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35496 deadline: 1680281803669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:43,671 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680281743666, max timestamp=1680281743669Num uniques 859; 2023-03-31 16:55:43,671 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680281743666, max timestamp=1680281743669Num uniques 859; 2023-03-31 16:55:43,671 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 16:55:43,671 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680281743666, max timestamp=1680281743669Num uniques 859; 2023-03-31 16:55:43,671 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 16:55:43,671 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 16:55:43,675 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=11 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/A/daf61565fc3f4daca4e22f65cc78999d 2023-03-31 16:55:43,692 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=11 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/B/d7a06737ddc9484f8e960bb281027c0d 2023-03-31 16:55:43,702 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=11 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/C/ccb45276e3cc44bdb5f0d7c25af6246f 2023-03-31 16:55:43,706 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/A/daf61565fc3f4daca4e22f65cc78999d as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/daf61565fc3f4daca4e22f65cc78999d 2023-03-31 16:55:43,710 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/daf61565fc3f4daca4e22f65cc78999d, entries=200, sequenceid=11, filesize=13.9 K 2023-03-31 16:55:43,710 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/B/d7a06737ddc9484f8e960bb281027c0d as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/d7a06737ddc9484f8e960bb281027c0d 2023-03-31 16:55:43,713 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/d7a06737ddc9484f8e960bb281027c0d, entries=150, sequenceid=11, filesize=11.6 K 2023-03-31 16:55:43,713 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/C/ccb45276e3cc44bdb5f0d7c25af6246f as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/ccb45276e3cc44bdb5f0d7c25af6246f 2023-03-31 16:55:43,716 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/ccb45276e3cc44bdb5f0d7c25af6246f, entries=150, sequenceid=11, filesize=11.6 K 2023-03-31 16:55:43,721 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=93.93 KB/96180 for fd96548248d41c209e12fca5d7040af5 in 86ms, sequenceid=11, compaction requested=false 2023-03-31 16:55:43,721 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fd96548248d41c209e12fca5d7040af5: 2023-03-31 16:55:43,721 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:43,721 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:43,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on fd96548248d41c209e12fca5d7040af5 2023-03-31 16:55:43,721 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing fd96548248d41c209e12fca5d7040af5 3/3 column families, dataSize=107.34 KB heapSize=305.30 KB 2023-03-31 16:55:43,721 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:43,721 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:43,721 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:43,721 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=A 2023-03-31 16:55:43,721 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:43,721 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:55:43,721 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=B 2023-03-31 16:55:43,721 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:55:43,722 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=C 2023-03-31 16:55:43,722 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:55:43,728 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:43,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:43,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:43,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:43,729 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:43,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:43,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:43,729 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:43,729 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:43,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:43,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:43,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:43,730 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:43,731 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@480b40cc[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:43,731 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:43,731 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:43,733 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:43,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:43,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:43,733 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:43,733 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:43,733 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:43,733 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:43,733 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:43,734 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:43,734 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:43,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:43,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:43,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:43,734 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:43,735 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:43,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:43,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:43,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:43,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:43,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:43,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:43,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:43,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:43,737 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:43,737 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:43,737 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:43,737 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:43,737 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:43,737 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:43,739 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:43,739 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:43,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35494 deadline: 1680281803739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:43,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:43,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:43,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:43,739 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:43,740 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:43,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:43,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:43,740 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:43,740 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:43,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:43,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:43,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:43,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35546 deadline: 1680281803740, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:43,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:43,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:43,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:43,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:43,742 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:43,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:43,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:43,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:43,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:43,744 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:43,744 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:43,744 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:43,744 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:43,744 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:43,744 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:43,744 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:43,744 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:43,744 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:43,744 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:43,745 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:43,745 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:43,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:43,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:43,744 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:43,744 DEBUG [Thread-3278] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:43,745 DEBUG [Thread-3278] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:43,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:43,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:43,753 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=44 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/A/51096b5d8f38425e965ecdb0dedda48a 2023-03-31 16:55:43,772 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:43,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35582 deadline: 1680281803772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:43,773 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:43,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35580 deadline: 1680281803772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:43,773 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:43,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35496 deadline: 1680281803772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:43,844 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:43,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35494 deadline: 1680281803841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:43,844 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:43,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35546 deadline: 1680281803842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:43,845 DEBUG [Thread-3278] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:43,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:43,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:43,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:43,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:43,849 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:43,849 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:43,849 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:43,849 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:43,849 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:43,849 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:43,852 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:43,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:43,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:43,852 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:43,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:43,852 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:43,852 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:43,853 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:43,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:43,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:43,853 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:43,860 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:43,860 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:43,860 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:43,860 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:43,860 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:43,860 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:43,860 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:43,860 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:43,861 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. started... 2023-03-31 16:55:43,861 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. as already flushing 2023-03-31 16:55:43,861 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:43,861 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:43,861 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:43,861 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:43,861 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:43,861 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:43,863 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:43,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:43,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:43,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:43,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:43,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:43,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:43,864 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:43,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:43,864 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:43,864 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:43,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:43,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:43,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c4c70e4[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:43,865 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:43,865 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:43,866 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:43,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:43,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:43,866 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:43,866 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:43,866 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:43,866 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:43,866 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:43,872 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:43,872 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:43,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:43,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:43,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:43,872 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:43,872 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:43,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:43,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:43,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:43,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:43,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:43,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:43,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:43,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:43,874 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:43,874 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:43,874 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:43,874 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:43,874 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:43,874 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:43,875 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:43,876 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:43,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:43,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:43,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:43,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:43,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:43,876 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:43,876 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:43,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:43,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:43,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:43,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:43,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:43,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:43,877 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:43,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:43,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:43,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:43,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:43,880 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:43,880 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:43,880 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:43,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:43,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:43,880 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:43,880 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:43,880 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:43,880 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:43,880 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:43,880 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:43,880 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:43,880 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:43,880 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:43,880 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:43,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:43,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:43,881 DEBUG [Thread-3278] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:43,881 DEBUG [Thread-3278] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:43,980 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:43,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35582 deadline: 1680281803974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:43,980 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:43,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35580 deadline: 1680281803975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:43,981 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:43,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35496 deadline: 1680281803975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:43,981 DEBUG [Thread-3278] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:43,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:43,983 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:43,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:43,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:43,988 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:43,988 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:43,988 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:43,988 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:43,988 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:43,988 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:43,990 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:43,990 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:43,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:43,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:43,990 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:43,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:43,990 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:43,991 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:43,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:43,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:43,991 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:43,991 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:43,991 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:43,991 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:43,991 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:43,991 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:43,991 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:43,991 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:43,991 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:43,992 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. started... 2023-03-31 16:55:43,992 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. as already flushing 2023-03-31 16:55:43,992 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:43,992 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:43,992 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:43,992 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:43,992 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:43,992 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:43,994 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:43,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:43,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:43,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:43,994 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:43,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:43,994 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:43,994 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:43,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:43,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:43,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:43,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:43,996 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:43,996 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f4d4fe6[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:43,996 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:43,996 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:43,997 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:43,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:43,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:43,997 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:43,997 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:43,997 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:43,997 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:43,997 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:43,998 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:43,998 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:43,998 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:43,998 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:43,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:43,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:43,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:43,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:43,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:43,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:44,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:44,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:44,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:44,001 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:44,001 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:44,001 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:44,001 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:44,001 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:44,001 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:44,002 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:44,003 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:44,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:44,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:44,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:44,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:44,003 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:44,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:44,003 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:44,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:44,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:44,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:44,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:44,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:44,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:44,004 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:44,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:44,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,008 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,008 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:44,008 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:44,008 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:44,008 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:44,008 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:44,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:44,009 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:44,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:44,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:44,008 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:44,008 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,009 DEBUG [Thread-3278] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:44,009 DEBUG [Thread-3278] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:44,009 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:44,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:44,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:44,009 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:44,009 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:44,048 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:44,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35494 deadline: 1680281804045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,048 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:44,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35546 deadline: 1680281804046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,109 DEBUG [Thread-3278] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:44,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:44,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:44,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:44,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:44,114 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:44,114 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:44,114 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:44,114 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:44,114 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:44,114 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:44,116 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,116 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:44,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:44,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:44,116 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,116 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:44,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:44,117 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:44,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:44,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:44,117 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:44,117 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:44,117 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:44,117 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:44,117 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:44,117 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:44,118 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:44,118 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:44,118 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. started... 2023-03-31 16:55:44,118 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:44,118 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. as already flushing 2023-03-31 16:55:44,118 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:44,118 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:44,118 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:44,118 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,118 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:44,118 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:44,119 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,119 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:44,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:44,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:44,120 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:44,120 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:44,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:44,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:44,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:44,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:44,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:44,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4eb9486f[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:44,121 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:44,121 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:44,123 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:44,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:44,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:44,123 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:44,123 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:44,123 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:44,123 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,123 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:44,126 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:44,126 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:44,126 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,126 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:44,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:44,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:44,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:44,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:44,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:44,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:44,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:44,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:44,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:44,128 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:44,128 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:44,128 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:44,128 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:44,128 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:44,130 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:44,130 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:44,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:44,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:44,130 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:44,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:44,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:44,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:44,130 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:44,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:44,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:44,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:44,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:44,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:44,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:44,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,132 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:44,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:44,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,133 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,134 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:44,134 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:44,134 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,134 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:44,134 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:44,134 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:44,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:44,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:44,134 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:44,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:44,134 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:44,134 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:44,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:44,134 DEBUG [Thread-3278] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:44,134 DEBUG [Thread-3278] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:44,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:44,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:44,182 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=44 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/B/71de6e7c03bf4dd386c62fe1369efaf8 2023-03-31 16:55:44,192 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=44 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/C/d7cd4d3f7efb4531b652c72ef9a96f20 2023-03-31 16:55:44,195 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/A/51096b5d8f38425e965ecdb0dedda48a as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/51096b5d8f38425e965ecdb0dedda48a 2023-03-31 16:55:44,198 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/51096b5d8f38425e965ecdb0dedda48a, entries=250, sequenceid=44, filesize=16.2 K 2023-03-31 16:55:44,199 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/B/71de6e7c03bf4dd386c62fe1369efaf8 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/71de6e7c03bf4dd386c62fe1369efaf8 2023-03-31 16:55:44,202 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/71de6e7c03bf4dd386c62fe1369efaf8, entries=150, sequenceid=44, filesize=11.6 K 2023-03-31 16:55:44,202 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/C/d7cd4d3f7efb4531b652c72ef9a96f20 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/d7cd4d3f7efb4531b652c72ef9a96f20 2023-03-31 16:55:44,205 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/d7cd4d3f7efb4531b652c72ef9a96f20, entries=150, sequenceid=44, filesize=11.6 K 2023-03-31 16:55:44,206 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~305.25 KB/312576, currentSize=73.80 KB/75570 for fd96548248d41c209e12fca5d7040af5 in 485ms, sequenceid=44, compaction requested=false 2023-03-31 16:55:44,206 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for fd96548248d41c209e12fca5d7040af5: 2023-03-31 16:55:44,234 DEBUG [Thread-3278] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:44,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:44,236 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:44,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:44,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:44,237 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:44,237 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:44,237 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:44,238 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:44,238 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:44,238 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:44,239 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:44,239 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:44,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:44,240 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,240 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:44,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:44,240 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:44,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:44,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:44,240 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:44,241 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:44,241 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:44,241 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:44,241 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:44,241 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:44,241 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:44,241 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:44,241 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:44,241 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. started... 2023-03-31 16:55:44,241 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fd96548248d41c209e12fca5d7040af5 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-31 16:55:44,241 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=A 2023-03-31 16:55:44,241 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:44,242 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=B 2023-03-31 16:55:44,242 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:44,242 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=C 2023-03-31 16:55:44,242 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:44,249 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/A/d184f08a38ab4d9c87a2587b0625f704 2023-03-31 16:55:44,259 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/B/a8c313e2aa4e48f693d9d3c03dc53c27 2023-03-31 16:55:44,276 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/C/7e094804cabc48b69e71f77925e5c460 2023-03-31 16:55:44,281 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/A/d184f08a38ab4d9c87a2587b0625f704 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/d184f08a38ab4d9c87a2587b0625f704 2023-03-31 16:55:44,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on fd96548248d41c209e12fca5d7040af5 2023-03-31 16:55:44,284 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. as already flushing 2023-03-31 16:55:44,284 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/d184f08a38ab4d9c87a2587b0625f704, entries=150, sequenceid=58, filesize=11.6 K 2023-03-31 16:55:44,285 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/B/a8c313e2aa4e48f693d9d3c03dc53c27 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/a8c313e2aa4e48f693d9d3c03dc53c27 2023-03-31 16:55:44,288 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/a8c313e2aa4e48f693d9d3c03dc53c27, entries=150, sequenceid=58, filesize=11.6 K 2023-03-31 16:55:44,289 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/C/7e094804cabc48b69e71f77925e5c460 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/7e094804cabc48b69e71f77925e5c460 2023-03-31 16:55:44,291 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/7e094804cabc48b69e71f77925e5c460, entries=150, sequenceid=58, filesize=11.6 K 2023-03-31 16:55:44,292 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=26.84 KB/27480 for fd96548248d41c209e12fca5d7040af5 in 51ms, sequenceid=58, compaction requested=true 2023-03-31 16:55:44,292 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fd96548248d41c209e12fca5d7040af5: 2023-03-31 16:55:44,292 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:44,292 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:44,292 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:44,292 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,292 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:44,292 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:44,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on fd96548248d41c209e12fca5d7040af5 2023-03-31 16:55:44,294 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing fd96548248d41c209e12fca5d7040af5 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-31 16:55:44,294 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=A 2023-03-31 16:55:44,294 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:44,294 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=B 2023-03-31 16:55:44,294 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:44,294 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=C 2023-03-31 16:55:44,294 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:44,296 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,296 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:44,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:44,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:44,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:44,297 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:44,297 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:44,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:44,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:44,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:44,299 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:44,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ca66c54[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:44,300 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:44,300 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=68 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/A/5a1b5b8cb60e476d9fe4141b68065b00 2023-03-31 16:55:44,300 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:44,301 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:44,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:44,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:44,301 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:44,301 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:44,301 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:44,301 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,302 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:44,302 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:44,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:44,303 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:44,303 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:44,303 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:44,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:44,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:44,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:44,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:44,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:44,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:44,307 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:44,307 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:44,307 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:44,307 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:44,307 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:44,307 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:44,313 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:44,313 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:44,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:44,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:44,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:44,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:44,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:44,313 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:44,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:44,313 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:44,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:44,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:44,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:44,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:44,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:44,315 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:44,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:44,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,317 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,317 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:44,317 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:44,317 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:44,317 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:44,317 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:44,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:44,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:44,317 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:44,317 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:44,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:44,317 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:44,317 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:44,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:44,317 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:44,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:44,317 DEBUG [Thread-3278] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:44,317 DEBUG [Thread-3278] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:44,339 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680281744324, max timestamp=1680281744337Num uniques 859; 2023-03-31 16:55:44,339 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680281744324, max timestamp=1680281744337Num uniques 859; 2023-03-31 16:55:44,339 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680281744324, max timestamp=1680281744337Num uniques 859; 2023-03-31 16:55:44,339 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 16:55:44,339 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 16:55:44,339 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 16:55:44,372 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:44,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35546 deadline: 1680281804364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,373 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:44,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35580 deadline: 1680281804364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,375 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:44,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35582 deadline: 1680281804371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,379 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:44,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35496 deadline: 1680281804372, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,380 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:44,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35494 deadline: 1680281804373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,418 DEBUG [Thread-3278] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:44,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:44,420 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:44,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:44,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:44,421 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:44,421 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:44,421 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:44,421 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:44,421 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:44,421 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:44,423 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,423 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:44,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:44,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:44,423 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,423 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:44,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:44,423 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:44,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:44,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:44,424 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:44,424 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:44,424 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:44,424 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:44,424 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:44,424 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:44,424 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:44,424 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:44,424 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:44,425 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. started... 2023-03-31 16:55:44,425 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. as already flushing 2023-03-31 16:55:44,425 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:44,425 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:44,425 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:44,425 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,425 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:44,425 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:44,426 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:44,426 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:44,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:44,427 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:44,427 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:44,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:44,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:44,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:44,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:44,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:44,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73c2fc74[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:44,428 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:44,428 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:44,430 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:44,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:44,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:44,430 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:44,430 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:44,430 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,430 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:44,430 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:44,432 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,432 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:44,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:44,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:44,432 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:44,432 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:44,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:44,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:44,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:44,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:44,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:44,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:44,434 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:44,434 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:44,435 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:44,435 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:44,435 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:44,435 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:44,436 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:44,436 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:44,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:44,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:44,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:44,436 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:44,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:44,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:44,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:44,437 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:44,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:44,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:44,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:44,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:44,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:44,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,438 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:44,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:44,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,440 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,440 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:44,440 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:44,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:44,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:44,440 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:44,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:44,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:44,440 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:44,440 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:44,440 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,440 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:44,440 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:44,440 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:44,440 DEBUG [Thread-3278] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:44,441 DEBUG [Thread-3278] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:44,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:44,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:44,478 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:44,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35546 deadline: 1680281804473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,478 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:44,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35580 deadline: 1680281804474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,481 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:44,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35582 deadline: 1680281804476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,486 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:44,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35496 deadline: 1680281804480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,486 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:44,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35494 deadline: 1680281804481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,541 DEBUG [Thread-3278] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:44,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:44,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:44,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:44,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:44,555 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:44,555 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:44,555 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:44,555 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:44,556 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:44,556 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:44,558 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:44,558 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:44,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:44,558 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,558 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:44,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:44,559 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:44,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:44,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:44,559 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:44,559 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:44,559 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:44,559 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:44,559 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:44,560 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:44,560 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:44,560 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:44,560 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:44,560 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. started... 2023-03-31 16:55:44,561 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. as already flushing 2023-03-31 16:55:44,561 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:44,561 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:44,561 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:44,561 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,561 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:44,561 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:44,563 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:44,563 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,563 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:44,563 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:44,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:44,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:44,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:44,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:44,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:44,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:44,570 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:44,570 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3635af95[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:44,570 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:44,570 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:44,572 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:44,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:44,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:44,572 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:44,572 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:44,572 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:44,572 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,572 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:44,574 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:44,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:44,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:44,575 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:44,575 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:44,575 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:44,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:44,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:44,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:44,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:44,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:44,578 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:44,578 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:44,578 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:44,578 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:44,578 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:44,578 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:44,580 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:44,580 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:44,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:44,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:44,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:44,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:44,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:44,580 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:44,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:44,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:44,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:44,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:44,582 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:44,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:44,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:44,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,583 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:44,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:44,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,586 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:44,586 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:44,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:44,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:44,586 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,586 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:44,586 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:44,586 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,586 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:44,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:44,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:44,586 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:44,586 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:44,586 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:44,586 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:44,586 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:44,587 DEBUG [Thread-3278] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:44,587 DEBUG [Thread-3278] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:44,686 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:44,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35546 deadline: 1680281804679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,686 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:44,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35580 deadline: 1680281804680, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,687 DEBUG [Thread-3278] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:44,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:44,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:44,689 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:44,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35582 deadline: 1680281804682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,689 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:44,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35496 deadline: 1680281804687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,690 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:44,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35494 deadline: 1680281804687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:44,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:44,690 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:44,690 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:44,690 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:44,690 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:44,690 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:44,691 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:44,692 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:44,692 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:44,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:44,692 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,692 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:44,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:44,692 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:44,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:44,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:44,693 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:44,693 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:44,693 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:44,693 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:44,693 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:44,693 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:44,693 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:44,693 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:44,693 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:44,694 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. started... 2023-03-31 16:55:44,694 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. as already flushing 2023-03-31 16:55:44,694 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:44,694 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:44,694 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:44,694 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,694 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:44,694 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:44,695 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,695 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:44,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:44,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:44,695 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:44,695 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:44,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:44,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:44,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:44,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:44,697 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:44,697 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@324b630c[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:44,697 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:44,697 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:44,699 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:44,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:44,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:44,700 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:44,700 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:44,700 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:44,700 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,700 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:44,701 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:44,701 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:44,701 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,701 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:44,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:44,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:44,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:44,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:44,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:44,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:44,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:44,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:44,703 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:44,703 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:44,703 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:44,703 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:44,703 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:44,703 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:44,705 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:44,705 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:44,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:44,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:44,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:44,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:44,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:44,705 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:44,705 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:44,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:44,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:44,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:44,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:44,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:44,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:44,706 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:44,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:44,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,709 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=68 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/B/aaeab0834e474f02bd12099b13627d9a 2023-03-31 16:55:44,712 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:44,712 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:44,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:44,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:44,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:44,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:44,714 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,714 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:44,714 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:44,714 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:44,714 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:44,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:44,715 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:44,715 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:44,715 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:44,715 DEBUG [Thread-3278] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:44,715 DEBUG [Thread-3278] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:44,715 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:44,718 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=68 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/C/5cb23aca555645fba28db8a653d0813b 2023-03-31 16:55:44,721 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/A/5a1b5b8cb60e476d9fe4141b68065b00 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/5a1b5b8cb60e476d9fe4141b68065b00 2023-03-31 16:55:44,726 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/5a1b5b8cb60e476d9fe4141b68065b00, entries=250, sequenceid=68, filesize=16.2 K 2023-03-31 16:55:44,727 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/B/aaeab0834e474f02bd12099b13627d9a as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/aaeab0834e474f02bd12099b13627d9a 2023-03-31 16:55:44,734 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/aaeab0834e474f02bd12099b13627d9a, entries=150, sequenceid=68, filesize=11.6 K 2023-03-31 16:55:44,736 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/C/5cb23aca555645fba28db8a653d0813b as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/5cb23aca555645fba28db8a653d0813b 2023-03-31 16:55:44,741 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/5cb23aca555645fba28db8a653d0813b, entries=150, sequenceid=68, filesize=11.6 K 2023-03-31 16:55:44,742 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for fd96548248d41c209e12fca5d7040af5 in 448ms, sequenceid=68, compaction requested=true 2023-03-31 16:55:44,742 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for fd96548248d41c209e12fca5d7040af5: 2023-03-31 16:55:44,742 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 16:55:44,742 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:55:44,742 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 16:55:44,742 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:55:44,742 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:55:44,746 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 59244 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 16:55:44,746 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1912): fd96548248d41c209e12fca5d7040af5/A is initiating minor compaction (all files) 2023-03-31 16:55:44,746 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of fd96548248d41c209e12fca5d7040af5/A in TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:44,746 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/daf61565fc3f4daca4e22f65cc78999d, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/51096b5d8f38425e965ecdb0dedda48a, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/d184f08a38ab4d9c87a2587b0625f704, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/5a1b5b8cb60e476d9fe4141b68065b00] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp, totalSize=57.9 K 2023-03-31 16:55:44,747 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting daf61565fc3f4daca4e22f65cc78999d, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1680281743628 2023-03-31 16:55:44,747 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 16:55:44,747 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): fd96548248d41c209e12fca5d7040af5/B is initiating minor compaction (all files) 2023-03-31 16:55:44,747 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fd96548248d41c209e12fca5d7040af5/B in TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:44,747 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/d7a06737ddc9484f8e960bb281027c0d, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/71de6e7c03bf4dd386c62fe1369efaf8, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/a8c313e2aa4e48f693d9d3c03dc53c27, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/aaeab0834e474f02bd12099b13627d9a] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp, totalSize=46.4 K 2023-03-31 16:55:44,747 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 51096b5d8f38425e965ecdb0dedda48a, keycount=250, bloomtype=ROW, size=16.2 K, encoding=NONE, compression=NONE, seqNum=44, earliestPutTs=1680281743681 2023-03-31 16:55:44,748 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting d7a06737ddc9484f8e960bb281027c0d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1680281743628 2023-03-31 16:55:44,748 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting d184f08a38ab4d9c87a2587b0625f704, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1680281743730 2023-03-31 16:55:44,748 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 71de6e7c03bf4dd386c62fe1369efaf8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=44, earliestPutTs=1680281743681 2023-03-31 16:55:44,750 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 5a1b5b8cb60e476d9fe4141b68065b00, keycount=250, bloomtype=ROW, size=16.2 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1680281744283 2023-03-31 16:55:44,750 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting a8c313e2aa4e48f693d9d3c03dc53c27, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1680281743730 2023-03-31 16:55:44,752 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting aaeab0834e474f02bd12099b13627d9a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1680281744283 2023-03-31 16:55:44,765 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] throttle.PressureAwareThroughputController(145): fd96548248d41c209e12fca5d7040af5#A#compaction#440 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:55:44,765 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): fd96548248d41c209e12fca5d7040af5#B#compaction#441 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:55:44,782 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/A/94e1a482c06f44fd8377ad9b367d1a7c as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/94e1a482c06f44fd8377ad9b367d1a7c 2023-03-31 16:55:44,783 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/B/c300e3d5f7404822a98b0ed4da06b338 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/c300e3d5f7404822a98b0ed4da06b338 2023-03-31 16:55:44,787 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in fd96548248d41c209e12fca5d7040af5/A of fd96548248d41c209e12fca5d7040af5 into 94e1a482c06f44fd8377ad9b367d1a7c(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:55:44,788 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for fd96548248d41c209e12fca5d7040af5: 2023-03-31 16:55:44,788 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5., storeName=fd96548248d41c209e12fca5d7040af5/A, priority=12, startTime=1680281744742; duration=0sec 2023-03-31 16:55:44,788 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:55:44,788 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 16:55:44,788 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in fd96548248d41c209e12fca5d7040af5/B of fd96548248d41c209e12fca5d7040af5 into c300e3d5f7404822a98b0ed4da06b338(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:55:44,788 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fd96548248d41c209e12fca5d7040af5: 2023-03-31 16:55:44,788 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5., storeName=fd96548248d41c209e12fca5d7040af5/B, priority=12, startTime=1680281744742; duration=0sec 2023-03-31 16:55:44,788 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:55:44,789 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 16:55:44,789 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1912): fd96548248d41c209e12fca5d7040af5/C is initiating minor compaction (all files) 2023-03-31 16:55:44,789 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of fd96548248d41c209e12fca5d7040af5/C in TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:44,789 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/ccb45276e3cc44bdb5f0d7c25af6246f, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/d7cd4d3f7efb4531b652c72ef9a96f20, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/7e094804cabc48b69e71f77925e5c460, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/5cb23aca555645fba28db8a653d0813b] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp, totalSize=46.4 K 2023-03-31 16:55:44,789 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting ccb45276e3cc44bdb5f0d7c25af6246f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1680281743628 2023-03-31 16:55:44,790 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting d7cd4d3f7efb4531b652c72ef9a96f20, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=44, earliestPutTs=1680281743681 2023-03-31 16:55:44,790 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 7e094804cabc48b69e71f77925e5c460, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1680281743730 2023-03-31 16:55:44,790 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 5cb23aca555645fba28db8a653d0813b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1680281744283 2023-03-31 16:55:44,797 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] throttle.PressureAwareThroughputController(145): fd96548248d41c209e12fca5d7040af5#C#compaction#442 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:55:44,815 DEBUG [Thread-3278] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:44,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:44,817 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:44,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:44,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:44,818 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:44,818 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:44,818 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:44,818 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:44,818 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:44,818 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:44,819 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:44,819 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:44,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:44,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:44,820 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,820 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:44,820 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:44,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:44,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:44,821 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:44,822 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:44,822 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:44,822 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:44,822 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:44,822 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:44,822 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:44,822 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:44,822 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:44,822 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. started... 2023-03-31 16:55:44,822 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fd96548248d41c209e12fca5d7040af5 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-03-31 16:55:44,823 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=A 2023-03-31 16:55:44,823 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:55:44,823 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=B 2023-03-31 16:55:44,823 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:55:44,823 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=C 2023-03-31 16:55:44,823 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:55:44,829 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=106 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/A/ba96916252d646d6bdfaf90176c04f05 2023-03-31 16:55:44,840 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=106 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/B/c6883da5c49848d1879d316b019fd300 2023-03-31 16:55:44,851 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=106 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/C/9899463f779d4f01b9faaf75b7339b58 2023-03-31 16:55:44,854 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/A/ba96916252d646d6bdfaf90176c04f05 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/ba96916252d646d6bdfaf90176c04f05 2023-03-31 16:55:44,857 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/ba96916252d646d6bdfaf90176c04f05, entries=150, sequenceid=106, filesize=11.6 K 2023-03-31 16:55:44,858 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/B/c6883da5c49848d1879d316b019fd300 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/c6883da5c49848d1879d316b019fd300 2023-03-31 16:55:44,861 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/c6883da5c49848d1879d316b019fd300, entries=150, sequenceid=106, filesize=11.6 K 2023-03-31 16:55:44,861 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/C/9899463f779d4f01b9faaf75b7339b58 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/9899463f779d4f01b9faaf75b7339b58 2023-03-31 16:55:44,864 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/9899463f779d4f01b9faaf75b7339b58, entries=150, sequenceid=106, filesize=11.6 K 2023-03-31 16:55:44,865 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=0 B/0 for fd96548248d41c209e12fca5d7040af5 in 43ms, sequenceid=106, compaction requested=false 2023-03-31 16:55:44,865 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fd96548248d41c209e12fca5d7040af5: 2023-03-31 16:55:44,865 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:44,865 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:44,865 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:44,865 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,865 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:44,865 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:44,866 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,866 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:44,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:44,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:44,867 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:44,867 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:44,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:44,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:44,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:44,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:44,868 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:44,868 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5de07ddd[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:44,868 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:44,868 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:44,870 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:44,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:44,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:44,870 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:44,870 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:44,870 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:44,872 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:44,872 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:44,872 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,872 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:44,873 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:44,873 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:44,873 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:44,873 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:44,873 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:44,873 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:44,873 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:44,874 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:44,874 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:44,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:44,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:44,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:44,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:44,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:44,874 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:44,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:44,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:44,875 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:44,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:44,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:44,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:44,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:44,878 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:44,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:44,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,880 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,880 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:44,880 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:44,880 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:44,880 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,880 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:44,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:44,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:44,880 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:44,880 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:44,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:44,880 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:44,880 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:44,881 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:44,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:44,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:44,881 DEBUG [Thread-3278] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:44,881 DEBUG [Thread-3278] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:44,881 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:44,981 DEBUG [Thread-3278] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:44,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:44,983 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:44,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:44,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:44,984 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:44,984 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:44,984 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:44,984 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:44,985 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:44,985 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:44,986 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:44,986 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:44,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:44,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:44,987 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:44,987 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:44,987 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:44,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:44,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:44,989 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:44,990 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:44,990 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:44,990 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:44,990 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:44,990 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:44,990 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:44,990 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:44,990 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:44,990 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. started... 2023-03-31 16:55:44,991 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fd96548248d41c209e12fca5d7040af5 3/3 column families, dataSize=6.71 KB heapSize=20.67 KB 2023-03-31 16:55:44,991 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=A 2023-03-31 16:55:44,991 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:44,991 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=B 2023-03-31 16:55:44,991 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:44,991 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=C 2023-03-31 16:55:44,991 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:44,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on fd96548248d41c209e12fca5d7040af5 2023-03-31 16:55:44,997 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. as already flushing 2023-03-31 16:55:45,017 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680281745006, max timestamp=1680281745016Num uniques 859; 2023-03-31 16:55:45,017 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680281745006, max timestamp=1680281745016Num uniques 859; 2023-03-31 16:55:45,017 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-31 16:55:45,017 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-31 16:55:45,018 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680281745006, max timestamp=1680281745016Num uniques 859; 2023-03-31 16:55:45,018 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-31 16:55:45,047 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=418, cellSize=19138, totalHeapSize=43712, min timestamp=1680281745026, max timestamp=1680281745045Num uniques 1118; 2023-03-31 16:55:45,047 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=19138, suffix heap size=144240,new segment heap size=43712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=418 2023-03-31 16:55:45,047 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=418, cellSize=19138, totalHeapSize=43712, min timestamp=1680281745026, max timestamp=1680281745045Num uniques 1118; 2023-03-31 16:55:45,047 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=19138, suffix heap size=144240,new segment heap size=43712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=418 2023-03-31 16:55:45,048 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=418, cellSize=19138, totalHeapSize=43712, min timestamp=1680281745026, max timestamp=1680281745045Num uniques 1118; 2023-03-31 16:55:45,048 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=19138, suffix heap size=144240,new segment heap size=43712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=418 2023-03-31 16:55:45,076 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1680281745061, max timestamp=1680281745074Num uniques 1277; 2023-03-31 16:55:45,076 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=58482, new segment data size=12682, suffix heap size=160776,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1277, new segment cells count=277 2023-03-31 16:55:45,077 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1680281745061, max timestamp=1680281745074Num uniques 1277; 2023-03-31 16:55:45,077 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1680281745061, max timestamp=1680281745074Num uniques 1277; 2023-03-31 16:55:45,077 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=58482, new segment data size=12682, suffix heap size=160776,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1277, new segment cells count=277 2023-03-31 16:55:45,077 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=58482, new segment data size=12682, suffix heap size=160776,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1277, new segment cells count=277 2023-03-31 16:55:45,103 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1680281745089, max timestamp=1680281745101Num uniques 1136; 2023-03-31 16:55:45,103 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=13096, suffix heap size=146112,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=286 2023-03-31 16:55:45,104 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1680281745089, max timestamp=1680281745101Num uniques 1136; 2023-03-31 16:55:45,104 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=13096, suffix heap size=146112,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=286 2023-03-31 16:55:45,104 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1680281745089, max timestamp=1680281745101Num uniques 1136; 2023-03-31 16:55:45,104 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=13096, suffix heap size=146112,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=286 2023-03-31 16:55:45,135 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1680281745119, max timestamp=1680281745133Num uniques 1145; 2023-03-31 16:55:45,135 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=13510, suffix heap size=147048,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=295 2023-03-31 16:55:45,136 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1680281745119, max timestamp=1680281745133Num uniques 1145; 2023-03-31 16:55:45,136 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1680281745119, max timestamp=1680281745133Num uniques 1145; 2023-03-31 16:55:45,136 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=13510, suffix heap size=147048,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=295 2023-03-31 16:55:45,136 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=13510, suffix heap size=147048,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=295 2023-03-31 16:55:45,162 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1680281745140, max timestamp=1680281745160Num uniques 1154; 2023-03-31 16:55:45,162 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=11630, suffix heap size=147984,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=254 2023-03-31 16:55:45,163 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1680281745140, max timestamp=1680281745160Num uniques 1154; 2023-03-31 16:55:45,163 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1680281745140, max timestamp=1680281745160Num uniques 1154; 2023-03-31 16:55:45,163 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=11630, suffix heap size=147984,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=254 2023-03-31 16:55:45,163 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=11630, suffix heap size=147984,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=254 2023-03-31 16:55:45,196 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1680281745185, max timestamp=1680281745193Num uniques 1113; 2023-03-31 16:55:45,196 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=9748, suffix heap size=143720,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=213 2023-03-31 16:55:45,196 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1680281745185, max timestamp=1680281745193Num uniques 1113; 2023-03-31 16:55:45,196 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1680281745185, max timestamp=1680281745193Num uniques 1113; 2023-03-31 16:55:45,196 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=9748, suffix heap size=143720,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=213 2023-03-31 16:55:45,196 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=9748, suffix heap size=143720,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=213 2023-03-31 16:55:45,204 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/C/f1a52c170c6942ad8fcd34df8610daff as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/f1a52c170c6942ad8fcd34df8610daff 2023-03-31 16:55:45,208 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in fd96548248d41c209e12fca5d7040af5/C of fd96548248d41c209e12fca5d7040af5 into f1a52c170c6942ad8fcd34df8610daff(size=11.7 K), total size for store is 23.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:55:45,208 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for fd96548248d41c209e12fca5d7040af5: 2023-03-31 16:55:45,208 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5., storeName=fd96548248d41c209e12fca5d7040af5/C, priority=12, startTime=1680281744742; duration=0sec 2023-03-31 16:55:45,208 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:55:45,234 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=372, cellSize=17032, totalHeapSize=38928, min timestamp=1680281745199, max timestamp=1680281745232Num uniques 1072; 2023-03-31 16:55:45,234 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=17032, suffix heap size=139456,new segment heap size=38928 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=372 2023-03-31 16:55:45,235 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=372, cellSize=17032, totalHeapSize=38928, min timestamp=1680281745199, max timestamp=1680281745232Num uniques 1072; 2023-03-31 16:55:45,235 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=17032, suffix heap size=139456,new segment heap size=38928 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=372 2023-03-31 16:55:45,235 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=372, cellSize=17032, totalHeapSize=38928, min timestamp=1680281745199, max timestamp=1680281745232Num uniques 1072; 2023-03-31 16:55:45,235 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=17032, suffix heap size=139456,new segment heap size=38928 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=372 2023-03-31 16:55:45,259 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1680281745248, max timestamp=1680281745257Num uniques 1231; 2023-03-31 16:55:45,259 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:45,259 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=56376, new segment data size=12866, suffix heap size=155992,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1231, new segment cells count=281 2023-03-31 16:55:45,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35496 deadline: 1680281805255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:45,260 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1680281745248, max timestamp=1680281745257Num uniques 1231; 2023-03-31 16:55:45,260 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1680281745248, max timestamp=1680281745257Num uniques 1231; 2023-03-31 16:55:45,260 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=56376, new segment data size=12866, suffix heap size=155992,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1231, new segment cells count=281 2023-03-31 16:55:45,260 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=56376, new segment data size=12866, suffix heap size=155992,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1231, new segment cells count=281 2023-03-31 16:55:45,292 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1680281745277, max timestamp=1680281745289Num uniques 1140; 2023-03-31 16:55:45,292 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1680281745277, max timestamp=1680281745289Num uniques 1140; 2023-03-31 16:55:45,292 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=52210, new segment data size=10990, suffix heap size=146528,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=240 2023-03-31 16:55:45,292 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=52210, new segment data size=10990, suffix heap size=146528,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=240 2023-03-31 16:55:45,292 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1680281745277, max timestamp=1680281745289Num uniques 1140; 2023-03-31 16:55:45,292 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=52210, new segment data size=10990, suffix heap size=146528,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=240 2023-03-31 16:55:45,321 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=299, cellSize=13694, totalHeapSize=31336, min timestamp=1680281745299, max timestamp=1680281745318Num uniques 1099; 2023-03-31 16:55:45,321 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=13694, suffix heap size=142264,new segment heap size=31336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=299 2023-03-31 16:55:45,321 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=299, cellSize=13694, totalHeapSize=31336, min timestamp=1680281745299, max timestamp=1680281745318Num uniques 1099; 2023-03-31 16:55:45,321 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=299, cellSize=13694, totalHeapSize=31336, min timestamp=1680281745299, max timestamp=1680281745318Num uniques 1099; 2023-03-31 16:55:45,321 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=13694, suffix heap size=142264,new segment heap size=31336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=299 2023-03-31 16:55:45,321 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=13694, suffix heap size=142264,new segment heap size=31336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=299 2023-03-31 16:55:45,356 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:45,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35494 deadline: 1680281805349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:45,356 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:45,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35546 deadline: 1680281805349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:45,356 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1680281745341, max timestamp=1680281745353Num uniques 1158; 2023-03-31 16:55:45,356 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1680281745341, max timestamp=1680281745353Num uniques 1158; 2023-03-31 16:55:45,356 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=53030, new segment data size=7230, suffix heap size=148400,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1158, new segment cells count=158 2023-03-31 16:55:45,356 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1680281745341, max timestamp=1680281745353Num uniques 1158; 2023-03-31 16:55:45,356 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=53030, new segment data size=7230, suffix heap size=148400,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1158, new segment cells count=158 2023-03-31 16:55:45,356 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=53030, new segment data size=7230, suffix heap size=148400,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1158, new segment cells count=158 2023-03-31 16:55:45,388 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=267, cellSize=12222, totalHeapSize=28008, min timestamp=1680281745353, max timestamp=1680281745386Num uniques 1017; 2023-03-31 16:55:45,388 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=12222, suffix heap size=133736,new segment heap size=28008 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=267 2023-03-31 16:55:45,388 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=267, cellSize=12222, totalHeapSize=28008, min timestamp=1680281745353, max timestamp=1680281745386Num uniques 1017; 2023-03-31 16:55:45,388 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=267, cellSize=12222, totalHeapSize=28008, min timestamp=1680281745353, max timestamp=1680281745386Num uniques 1017; 2023-03-31 16:55:45,389 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=12222, suffix heap size=133736,new segment heap size=28008 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=267 2023-03-31 16:55:45,389 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=12222, suffix heap size=133736,new segment heap size=28008 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=267 2023-03-31 16:55:45,399 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=111 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/A/4161aa1256cf47f6badfcb397959f7e4 2023-03-31 16:55:45,426 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=111 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/B/dcee5968ec6d4db98315146693565e29 2023-03-31 16:55:45,432 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=276, cellSize=12636, totalHeapSize=28944, min timestamp=1680281745406, max timestamp=1680281745429Num uniques 1126; 2023-03-31 16:55:45,432 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=51566, new segment data size=12636, suffix heap size=145072,new segment heap size=28944 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1126, new segment cells count=276 2023-03-31 16:55:45,432 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=276, cellSize=12636, totalHeapSize=28944, min timestamp=1680281745406, max timestamp=1680281745429Num uniques 1126; 2023-03-31 16:55:45,432 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=51566, new segment data size=12636, suffix heap size=145072,new segment heap size=28944 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1126, new segment cells count=276 2023-03-31 16:55:45,433 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=276, cellSize=12636, totalHeapSize=28944, min timestamp=1680281745406, max timestamp=1680281745429Num uniques 1126; 2023-03-31 16:55:45,433 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=51566, new segment data size=12636, suffix heap size=145072,new segment heap size=28944 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1126, new segment cells count=276 2023-03-31 16:55:45,467 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=335, cellSize=15340, totalHeapSize=35080, min timestamp=1680281745455, max timestamp=1680281745465Num uniques 1135; 2023-03-31 16:55:45,467 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=51980, new segment data size=15340, suffix heap size=146008,new segment heap size=35080 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1135, new segment cells count=335 2023-03-31 16:55:45,468 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=335, cellSize=15340, totalHeapSize=35080, min timestamp=1680281745455, max timestamp=1680281745465Num uniques 1135; 2023-03-31 16:55:45,468 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=51980, new segment data size=15340, suffix heap size=146008,new segment heap size=35080 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1135, new segment cells count=335 2023-03-31 16:55:45,468 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=335, cellSize=15340, totalHeapSize=35080, min timestamp=1680281745455, max timestamp=1680281745465Num uniques 1135; 2023-03-31 16:55:45,468 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=51980, new segment data size=15340, suffix heap size=146008,new segment heap size=35080 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1135, new segment cells count=335 2023-03-31 16:55:45,495 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=244, cellSize=11174, totalHeapSize=25616, min timestamp=1680281745484, max timestamp=1680281745493Num uniques 1194; 2023-03-31 16:55:45,495 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=54684, new segment data size=11174, suffix heap size=152144,new segment heap size=25616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1194, new segment cells count=244 2023-03-31 16:55:45,495 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=244, cellSize=11174, totalHeapSize=25616, min timestamp=1680281745484, max timestamp=1680281745493Num uniques 1194; 2023-03-31 16:55:45,495 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=54684, new segment data size=11174, suffix heap size=152144,new segment heap size=25616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1194, new segment cells count=244 2023-03-31 16:55:45,495 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=244, cellSize=11174, totalHeapSize=25616, min timestamp=1680281745484, max timestamp=1680281745493Num uniques 1194; 2023-03-31 16:55:45,496 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=54684, new segment data size=11174, suffix heap size=152144,new segment heap size=25616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1194, new segment cells count=244 2023-03-31 16:55:45,523 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=303, cellSize=13874, totalHeapSize=31752, min timestamp=1680281745504, max timestamp=1680281745521Num uniques 1103; 2023-03-31 16:55:45,523 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50514, new segment data size=13874, suffix heap size=142680,new segment heap size=31752 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1103, new segment cells count=303 2023-03-31 16:55:45,524 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=303, cellSize=13868, totalHeapSize=31752, min timestamp=1680281745504, max timestamp=1680281745521Num uniques 1103; 2023-03-31 16:55:45,524 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50508, new segment data size=13868, suffix heap size=142680,new segment heap size=31752 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1103, new segment cells count=303 2023-03-31 16:55:45,524 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=303, cellSize=13871, totalHeapSize=31752, min timestamp=1680281745504, max timestamp=1680281745521Num uniques 1103; 2023-03-31 16:55:45,524 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50511, new segment data size=13871, suffix heap size=142680,new segment heap size=31752 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1103, new segment cells count=303 2023-03-31 16:55:45,592 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=162, cellSize=7412, totalHeapSize=17088, min timestamp=1680281745544, max timestamp=1680281745589Num uniques 1162; 2023-03-31 16:55:45,592 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=53212, new segment data size=7412, suffix heap size=148816,new segment heap size=17088 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1162, new segment cells count=162 2023-03-31 16:55:45,592 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=162, cellSize=7412, totalHeapSize=17088, min timestamp=1680281745544, max timestamp=1680281745589Num uniques 1162; 2023-03-31 16:55:45,592 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=53212, new segment data size=7412, suffix heap size=148816,new segment heap size=17088 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1162, new segment cells count=162 2023-03-31 16:55:45,594 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=162, cellSize=7412, totalHeapSize=17088, min timestamp=1680281745544, max timestamp=1680281745589Num uniques 1162; 2023-03-31 16:55:45,594 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=53212, new segment data size=7412, suffix heap size=148816,new segment heap size=17088 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1162, new segment cells count=162 2023-03-31 16:55:45,625 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=271, cellSize=12406, totalHeapSize=28424, min timestamp=1680281745611, max timestamp=1680281745623Num uniques 1021; 2023-03-31 16:55:45,625 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46756, new segment data size=12406, suffix heap size=134152,new segment heap size=28424 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1021, new segment cells count=271 2023-03-31 16:55:45,626 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=271, cellSize=12406, totalHeapSize=28424, min timestamp=1680281745611, max timestamp=1680281745623Num uniques 1021; 2023-03-31 16:55:45,626 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46756, new segment data size=12406, suffix heap size=134152,new segment heap size=28424 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1021, new segment cells count=271 2023-03-31 16:55:45,626 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=271, cellSize=12406, totalHeapSize=28424, min timestamp=1680281745611, max timestamp=1680281745623Num uniques 1021; 2023-03-31 16:55:45,626 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46756, new segment data size=12406, suffix heap size=134152,new segment heap size=28424 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1021, new segment cells count=271 2023-03-31 16:55:45,658 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=280, cellSize=12820, totalHeapSize=29360, min timestamp=1680281745637, max timestamp=1680281745655Num uniques 1130; 2023-03-31 16:55:45,658 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=51750, new segment data size=12820, suffix heap size=145488,new segment heap size=29360 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1130, new segment cells count=280 2023-03-31 16:55:45,658 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=280, cellSize=12820, totalHeapSize=29360, min timestamp=1680281745637, max timestamp=1680281745655Num uniques 1130; 2023-03-31 16:55:45,658 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=51750, new segment data size=12820, suffix heap size=145488,new segment heap size=29360 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1130, new segment cells count=280 2023-03-31 16:55:45,660 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=280, cellSize=12820, totalHeapSize=29360, min timestamp=1680281745637, max timestamp=1680281745655Num uniques 1130; 2023-03-31 16:55:45,661 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=51750, new segment data size=12820, suffix heap size=145488,new segment heap size=29360 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1130, new segment cells count=280 2023-03-31 16:55:45,690 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=339, cellSize=15524, totalHeapSize=35496, min timestamp=1680281745671, max timestamp=1680281745688Num uniques 1139; 2023-03-31 16:55:45,690 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=52164, new segment data size=15524, suffix heap size=146424,new segment heap size=35496 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1139, new segment cells count=339 2023-03-31 16:55:45,691 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=339, cellSize=15524, totalHeapSize=35496, min timestamp=1680281745671, max timestamp=1680281745688Num uniques 1139; 2023-03-31 16:55:45,691 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=52164, new segment data size=15524, suffix heap size=146424,new segment heap size=35496 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1139, new segment cells count=339 2023-03-31 16:55:45,691 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=339, cellSize=15524, totalHeapSize=35496, min timestamp=1680281745671, max timestamp=1680281745688Num uniques 1139; 2023-03-31 16:55:45,691 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=52164, new segment data size=15524, suffix heap size=146424,new segment heap size=35496 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1139, new segment cells count=339 2023-03-31 16:55:45,720 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=248, cellSize=11358, totalHeapSize=26032, min timestamp=1680281745698, max timestamp=1680281745717Num uniques 1198; 2023-03-31 16:55:45,720 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=248, cellSize=11358, totalHeapSize=26032, min timestamp=1680281745698, max timestamp=1680281745717Num uniques 1198; 2023-03-31 16:55:45,720 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=54868, new segment data size=11358, suffix heap size=152560,new segment heap size=26032 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1198, new segment cells count=248 2023-03-31 16:55:45,720 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=248, cellSize=11358, totalHeapSize=26032, min timestamp=1680281745698, max timestamp=1680281745717Num uniques 1198; 2023-03-31 16:55:45,720 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=54868, new segment data size=11358, suffix heap size=152560,new segment heap size=26032 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1198, new segment cells count=248 2023-03-31 16:55:45,720 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=54868, new segment data size=11358, suffix heap size=152560,new segment heap size=26032 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1198, new segment cells count=248 2023-03-31 16:55:45,749 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=357, cellSize=16345, totalHeapSize=37368, min timestamp=1680281745732, max timestamp=1680281745747Num uniques 1107; 2023-03-31 16:55:45,749 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50695, new segment data size=16345, suffix heap size=143096,new segment heap size=37368 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1107, new segment cells count=357 2023-03-31 16:55:45,749 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=357, cellSize=16345, totalHeapSize=37368, min timestamp=1680281745732, max timestamp=1680281745747Num uniques 1107; 2023-03-31 16:55:45,749 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50695, new segment data size=16345, suffix heap size=143096,new segment heap size=37368 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1107, new segment cells count=357 2023-03-31 16:55:45,750 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=357, cellSize=16345, totalHeapSize=37368, min timestamp=1680281745732, max timestamp=1680281745747Num uniques 1107; 2023-03-31 16:55:45,750 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50695, new segment data size=16345, suffix heap size=143096,new segment heap size=37368 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1107, new segment cells count=357 2023-03-31 16:55:45,778 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=216, cellSize=9886, totalHeapSize=22704, min timestamp=1680281745758, max timestamp=1680281745776Num uniques 1216; 2023-03-31 16:55:45,778 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=55686, new segment data size=9886, suffix heap size=154432,new segment heap size=22704 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1216, new segment cells count=216 2023-03-31 16:55:45,779 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=216, cellSize=9886, totalHeapSize=22704, min timestamp=1680281745758, max timestamp=1680281745776Num uniques 1216; 2023-03-31 16:55:45,779 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=55686, new segment data size=9886, suffix heap size=154432,new segment heap size=22704 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1216, new segment cells count=216 2023-03-31 16:55:45,779 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=216, cellSize=9886, totalHeapSize=22704, min timestamp=1680281745758, max timestamp=1680281745776Num uniques 1216; 2023-03-31 16:55:45,779 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=55686, new segment data size=9886, suffix heap size=154432,new segment heap size=22704 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1216, new segment cells count=216 2023-03-31 16:55:45,807 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=425, cellSize=19460, totalHeapSize=44440, min timestamp=1680281745779, max timestamp=1680281745804Num uniques 1075; 2023-03-31 16:55:45,807 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=425, cellSize=19460, totalHeapSize=44440, min timestamp=1680281745779, max timestamp=1680281745804Num uniques 1075; 2023-03-31 16:55:45,807 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49230, new segment data size=19460, suffix heap size=139768,new segment heap size=44440 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1075, new segment cells count=425 2023-03-31 16:55:45,807 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=425, cellSize=19460, totalHeapSize=44440, min timestamp=1680281745779, max timestamp=1680281745804Num uniques 1075; 2023-03-31 16:55:45,808 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49230, new segment data size=19460, suffix heap size=139768,new segment heap size=44440 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1075, new segment cells count=425 2023-03-31 16:55:45,807 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49230, new segment data size=19460, suffix heap size=139768,new segment heap size=44440 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1075, new segment cells count=425 2023-03-31 16:55:45,831 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:45,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35582 deadline: 1680281805828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:45,832 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=334, cellSize=15294, totalHeapSize=34976, min timestamp=1680281745819, max timestamp=1680281745830Num uniques 1284; 2023-03-31 16:55:45,833 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=58804, new segment data size=15294, suffix heap size=161504,new segment heap size=34976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1284, new segment cells count=334 2023-03-31 16:55:45,833 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=334, cellSize=15294, totalHeapSize=34976, min timestamp=1680281745819, max timestamp=1680281745830Num uniques 1284; 2023-03-31 16:55:45,833 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=58804, new segment data size=15294, suffix heap size=161504,new segment heap size=34976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1284, new segment cells count=334 2023-03-31 16:55:45,833 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=334, cellSize=15294, totalHeapSize=34976, min timestamp=1680281745819, max timestamp=1680281745830Num uniques 1284; 2023-03-31 16:55:45,833 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=58804, new segment data size=15294, suffix heap size=161504,new segment heap size=34976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1284, new segment cells count=334 2023-03-31 16:55:45,840 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=111 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/C/91677ed0537c4217b140719c20221425 2023-03-31 16:55:45,843 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/A/4161aa1256cf47f6badfcb397959f7e4 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/4161aa1256cf47f6badfcb397959f7e4 2023-03-31 16:55:45,846 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/4161aa1256cf47f6badfcb397959f7e4, entries=100, sequenceid=111, filesize=9.3 K 2023-03-31 16:55:45,847 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/B/dcee5968ec6d4db98315146693565e29 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/dcee5968ec6d4db98315146693565e29 2023-03-31 16:55:45,850 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/dcee5968ec6d4db98315146693565e29, entries=100, sequenceid=111, filesize=9.3 K 2023-03-31 16:55:45,850 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/C/91677ed0537c4217b140719c20221425 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/91677ed0537c4217b140719c20221425 2023-03-31 16:55:45,853 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/91677ed0537c4217b140719c20221425, entries=100, sequenceid=111, filesize=9.3 K 2023-03-31 16:55:45,856 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~13.42 KB/13740, heapSize ~40.55 KB/41520, currentSize=120.76 KB/123660 for fd96548248d41c209e12fca5d7040af5 in 866ms, sequenceid=111, compaction requested=true 2023-03-31 16:55:45,856 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fd96548248d41c209e12fca5d7040af5: 2023-03-31 16:55:45,856 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:45,856 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:45,856 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:45,856 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:45,856 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:45,856 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:45,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on fd96548248d41c209e12fca5d7040af5 2023-03-31 16:55:45,858 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing fd96548248d41c209e12fca5d7040af5 3/3 column families, dataSize=127.47 KB heapSize=348.66 KB 2023-03-31 16:55:45,858 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=A 2023-03-31 16:55:45,858 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:55:45,858 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=B 2023-03-31 16:55:45,858 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:55:45,858 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=C 2023-03-31 16:55:45,858 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:55:45,859 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:45,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:45,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:45,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:45,859 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:45,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:45,859 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:45,859 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:45,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:45,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:45,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:45,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:45,861 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:45,861 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39251b49[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:45,861 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:45,861 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:45,863 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:45,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:45,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:45,863 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:45,863 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:45,863 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:45,863 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:45,863 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:45,864 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:45,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:45,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:45,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:45,865 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:45,865 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:45,865 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:45,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:45,865 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=574 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/A/8c5edf7daad04a9e96b32cc8a516d5e3 2023-03-31 16:55:45,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:45,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:45,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:45,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:45,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:45,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:45,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:45,868 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:45,868 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:45,868 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:45,868 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:45,868 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:45,868 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:45,869 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:45,869 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:45,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:45,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:45,869 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:45,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:45,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:45,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:45,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:45,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:45,870 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:45,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:45,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:45,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:45,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:45,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:45,871 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:45,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:45,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:45,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:45,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:45,876 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:45,876 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:45,876 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:45,876 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:45,876 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:45,876 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:45,876 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:45,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:45,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:45,876 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:45,876 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:45,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:45,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:45,876 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:45,876 DEBUG [Thread-3278] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:45,876 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:45,876 DEBUG [Thread-3278] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:45,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:45,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:45,877 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:45,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35494 deadline: 1680281805872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:45,883 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=574 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/B/4f80bbf865894e77b40725b14fda87a8 2023-03-31 16:55:45,884 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:45,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35580 deadline: 1680281805873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:45,885 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:45,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35546 deadline: 1680281805874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:45,885 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:45,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35496 deadline: 1680281805877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:45,891 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=574 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/C/ae5fb4495bd0447d97f566cdc588ae6b 2023-03-31 16:55:45,894 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/A/8c5edf7daad04a9e96b32cc8a516d5e3 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/8c5edf7daad04a9e96b32cc8a516d5e3 2023-03-31 16:55:45,896 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/8c5edf7daad04a9e96b32cc8a516d5e3, entries=250, sequenceid=574, filesize=16.7 K 2023-03-31 16:55:45,897 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/B/4f80bbf865894e77b40725b14fda87a8 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/4f80bbf865894e77b40725b14fda87a8 2023-03-31 16:55:45,899 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/4f80bbf865894e77b40725b14fda87a8, entries=150, sequenceid=574, filesize=11.9 K 2023-03-31 16:55:45,900 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/C/ae5fb4495bd0447d97f566cdc588ae6b as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/ae5fb4495bd0447d97f566cdc588ae6b 2023-03-31 16:55:45,903 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/ae5fb4495bd0447d97f566cdc588ae6b, entries=150, sequenceid=574, filesize=11.9 K 2023-03-31 16:55:45,904 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~348.61 KB/356976, currentSize=60.38 KB/61830 for fd96548248d41c209e12fca5d7040af5 in 46ms, sequenceid=574, compaction requested=true 2023-03-31 16:55:45,904 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for fd96548248d41c209e12fca5d7040af5: 2023-03-31 16:55:45,904 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:55:45,904 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 16:55:45,904 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:55:45,904 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:55:45,904 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 16:55:45,905 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 50518 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 16:55:45,905 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45638 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 16:55:45,905 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1912): fd96548248d41c209e12fca5d7040af5/A is initiating minor compaction (all files) 2023-03-31 16:55:45,905 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): fd96548248d41c209e12fca5d7040af5/B is initiating minor compaction (all files) 2023-03-31 16:55:45,905 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of fd96548248d41c209e12fca5d7040af5/A in TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:45,905 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fd96548248d41c209e12fca5d7040af5/B in TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:45,905 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/94e1a482c06f44fd8377ad9b367d1a7c, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/ba96916252d646d6bdfaf90176c04f05, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/4161aa1256cf47f6badfcb397959f7e4, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/8c5edf7daad04a9e96b32cc8a516d5e3] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp, totalSize=49.3 K 2023-03-31 16:55:45,905 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/c300e3d5f7404822a98b0ed4da06b338, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/c6883da5c49848d1879d316b019fd300, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/dcee5968ec6d4db98315146693565e29, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/4f80bbf865894e77b40725b14fda87a8] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp, totalSize=44.6 K 2023-03-31 16:55:45,906 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 94e1a482c06f44fd8377ad9b367d1a7c, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1680281744283 2023-03-31 16:55:45,906 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting c300e3d5f7404822a98b0ed4da06b338, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1680281744283 2023-03-31 16:55:45,906 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting ba96916252d646d6bdfaf90176c04f05, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=106, earliestPutTs=1680281744362 2023-03-31 16:55:45,906 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting c6883da5c49848d1879d316b019fd300, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=106, earliestPutTs=1680281744362 2023-03-31 16:55:45,906 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 4161aa1256cf47f6badfcb397959f7e4, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=111, earliestPutTs=1680281744989 2023-03-31 16:55:45,906 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting dcee5968ec6d4db98315146693565e29, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=111, earliestPutTs=1680281744989 2023-03-31 16:55:45,906 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 8c5edf7daad04a9e96b32cc8a516d5e3, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=574, earliestPutTs=1680281745845 2023-03-31 16:55:45,906 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 4f80bbf865894e77b40725b14fda87a8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=574, earliestPutTs=1680281745848 2023-03-31 16:55:45,913 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] throttle.PressureAwareThroughputController(145): fd96548248d41c209e12fca5d7040af5#A#compaction#452 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:55:45,914 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): fd96548248d41c209e12fca5d7040af5#B#compaction#453 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:55:45,928 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/A/ac1a7331a0ee480abd994c5979aa85af as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/ac1a7331a0ee480abd994c5979aa85af 2023-03-31 16:55:45,932 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in fd96548248d41c209e12fca5d7040af5/A of fd96548248d41c209e12fca5d7040af5 into ac1a7331a0ee480abd994c5979aa85af(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:55:45,932 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for fd96548248d41c209e12fca5d7040af5: 2023-03-31 16:55:45,932 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5., storeName=fd96548248d41c209e12fca5d7040af5/A, priority=12, startTime=1680281745904; duration=0sec 2023-03-31 16:55:45,932 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:55:45,932 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 16:55:45,933 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45638 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 16:55:45,933 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1912): fd96548248d41c209e12fca5d7040af5/C is initiating minor compaction (all files) 2023-03-31 16:55:45,933 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of fd96548248d41c209e12fca5d7040af5/C in TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:45,933 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/f1a52c170c6942ad8fcd34df8610daff, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/9899463f779d4f01b9faaf75b7339b58, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/91677ed0537c4217b140719c20221425, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/ae5fb4495bd0447d97f566cdc588ae6b] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp, totalSize=44.6 K 2023-03-31 16:55:45,933 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting f1a52c170c6942ad8fcd34df8610daff, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1680281744283 2023-03-31 16:55:45,935 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 9899463f779d4f01b9faaf75b7339b58, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=106, earliestPutTs=1680281744362 2023-03-31 16:55:45,935 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 91677ed0537c4217b140719c20221425, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=111, earliestPutTs=1680281744989 2023-03-31 16:55:45,935 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting ae5fb4495bd0447d97f566cdc588ae6b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=574, earliestPutTs=1680281745848 2023-03-31 16:55:45,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on fd96548248d41c209e12fca5d7040af5 2023-03-31 16:55:45,937 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing fd96548248d41c209e12fca5d7040af5 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-31 16:55:45,937 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=A 2023-03-31 16:55:45,937 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:45,937 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=B 2023-03-31 16:55:45,937 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:45,937 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=C 2023-03-31 16:55:45,937 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:45,943 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] throttle.PressureAwareThroughputController(145): fd96548248d41c209e12fca5d7040af5#C#compaction#455 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:55:45,977 DEBUG [Thread-3278] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:45,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:45,978 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:45,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:45,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:45,979 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:45,979 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:45,979 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:45,979 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:45,980 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:45,980 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:45,981 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:45,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:45,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:45,981 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:45,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:45,981 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:45,982 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:45,982 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:45,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:45,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:45,982 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:45,982 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:45,982 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:45,982 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:45,982 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:45,982 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:45,983 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:45,983 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:45,983 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:45,983 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. started... 2023-03-31 16:55:45,983 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. as already flushing 2023-03-31 16:55:45,983 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:45,983 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:45,983 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:45,983 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:45,983 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:45,983 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:45,984 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:45,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:45,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:45,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:45,985 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:45,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:45,988 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:45,988 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:45,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:45,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:45,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:45,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:45,990 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:45,990 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22f752ff[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:45,990 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:45,990 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:45,992 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:45,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:45,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:45,992 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:45,992 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:45,992 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:45,992 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:45,992 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:45,994 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:45,995 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:45,995 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:45,995 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:45,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:45,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:45,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:45,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:45,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:45,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:45,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:45,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:45,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:45,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:45,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:45,998 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:45,998 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:45,998 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:45,998 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:45,998 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:45,998 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:45,999 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:45,999 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:46,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:46,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:46,000 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:46,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:46,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:46,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:46,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:46,000 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:46,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:46,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:46,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:46,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:46,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:46,001 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:46,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:46,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,003 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,003 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:46,003 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:46,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:46,003 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:46,003 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:46,003 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:46,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:46,003 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:46,004 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:46,003 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:46,004 DEBUG [Thread-3278] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:46,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:46,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:46,004 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,004 DEBUG [Thread-3278] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:46,004 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:46,004 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:46,004 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:46,006 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:46,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35494 deadline: 1680281805996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,009 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:46,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35496 deadline: 1680281805997, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,009 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:46,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35546 deadline: 1680281805997, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,009 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:46,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35582 deadline: 1680281805998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,009 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:46,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35580 deadline: 1680281805999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,104 DEBUG [Thread-3278] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:46,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:46,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:46,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:46,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:46,107 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:46,107 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:46,107 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:46,107 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:46,108 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:46,108 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:46,112 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:46,112 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:46,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:46,112 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,112 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:46,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:46,113 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:46,113 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:46,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:46,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:46,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35494 deadline: 1680281806107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,113 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:46,113 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:46,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35496 deadline: 1680281806110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,113 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:46,113 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:46,113 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:46,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35546 deadline: 1680281806110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,114 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:46,114 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:46,114 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:46,114 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:46,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35582 deadline: 1680281806110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,114 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:46,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35580 deadline: 1680281806110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,116 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:46,116 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:46,116 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:46,116 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. started... 2023-03-31 16:55:46,116 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. as already flushing 2023-03-31 16:55:46,116 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:46,116 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:46,116 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:46,116 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,116 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:46,116 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:46,119 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,119 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:46,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:46,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:46,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:46,119 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:46,119 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:46,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:46,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:46,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:46,120 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:46,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b4d16ae[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:46,121 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:46,121 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:46,122 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:46,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:46,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:46,122 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:46,122 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,122 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:46,122 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:46,122 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:46,123 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,123 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:46,124 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:46,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:46,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:46,124 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:46,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:46,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:46,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:46,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:46,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:46,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:46,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:46,126 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:46,126 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:46,126 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:46,126 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:46,126 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:46,128 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:46,128 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:46,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:46,128 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:46,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:46,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:46,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:46,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:46,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:46,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:46,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:46,129 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:46,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:46,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:46,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:46,130 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:46,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:46,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,133 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,133 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:46,133 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:46,133 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:46,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:46,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:46,133 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:46,133 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:46,133 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,133 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:46,133 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:46,133 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:46,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:46,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:46,133 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:46,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:46,134 DEBUG [Thread-3278] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:46,134 DEBUG [Thread-3278] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:46,234 DEBUG [Thread-3278] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:46,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:46,236 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:46,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:46,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:46,237 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:46,237 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:46,237 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:46,237 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:46,238 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:46,238 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:46,239 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:46,239 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:46,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:46,239 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,239 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:46,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:46,239 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:46,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:46,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:46,240 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:46,240 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:46,240 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:46,240 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:46,240 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:46,240 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:46,240 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:46,241 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:46,241 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:46,241 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. started... 2023-03-31 16:55:46,241 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. as already flushing 2023-03-31 16:55:46,241 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:46,241 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:46,241 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:46,241 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,241 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:46,241 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:46,242 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,242 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:46,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:46,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:46,243 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:46,243 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:46,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:46,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:46,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:46,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:46,244 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:46,244 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@159d848f[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:46,244 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:46,244 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:46,245 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:46,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:46,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:46,245 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:46,245 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:46,246 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:46,246 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,246 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:46,248 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,248 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:46,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:46,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:46,248 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:46,248 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:46,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:46,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:46,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:46,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:46,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:46,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:46,251 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:46,251 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:46,251 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:46,251 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:46,251 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:46,252 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:46,253 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:46,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:46,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:46,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:46,253 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:46,253 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:46,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:46,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:46,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:46,254 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:46,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:46,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:46,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:46,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:46,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:46,255 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:46,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:46,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,257 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,257 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:46,257 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:46,257 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:46,257 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,257 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:46,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:46,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:46,257 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:46,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:46,258 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:46,257 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:46,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:46,258 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:46,258 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:46,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:46,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:46,258 DEBUG [Thread-3278] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:46,258 DEBUG [Thread-3278] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:46,319 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:46,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35496 deadline: 1680281806314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,319 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:46,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35494 deadline: 1680281806315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,319 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:46,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35546 deadline: 1680281806315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,320 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:46,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35582 deadline: 1680281806316, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,321 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:46,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35580 deadline: 1680281806316, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,328 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/B/353ad655953f4c29bb1028c2855023ff as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/353ad655953f4c29bb1028c2855023ff 2023-03-31 16:55:46,331 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in fd96548248d41c209e12fca5d7040af5/B of fd96548248d41c209e12fca5d7040af5 into 353ad655953f4c29bb1028c2855023ff(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:55:46,332 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fd96548248d41c209e12fca5d7040af5: 2023-03-31 16:55:46,332 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5., storeName=fd96548248d41c209e12fca5d7040af5/B, priority=12, startTime=1680281745904; duration=0sec 2023-03-31 16:55:46,332 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:55:46,347 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=588 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/A/1af39476ae1e488794a4a84669d8b770 2023-03-31 16:55:46,353 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/C/9e45e98286c94a5dbd25d44026aa9e03 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/9e45e98286c94a5dbd25d44026aa9e03 2023-03-31 16:55:46,358 DEBUG [Thread-3278] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:46,358 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in fd96548248d41c209e12fca5d7040af5/C of fd96548248d41c209e12fca5d7040af5 into 9e45e98286c94a5dbd25d44026aa9e03(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:55:46,358 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for fd96548248d41c209e12fca5d7040af5: 2023-03-31 16:55:46,358 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5., storeName=fd96548248d41c209e12fca5d7040af5/C, priority=12, startTime=1680281745904; duration=0sec 2023-03-31 16:55:46,359 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:55:46,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:46,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:46,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:46,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:46,361 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:46,361 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:46,361 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:46,361 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:46,362 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:46,362 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:46,363 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:46,363 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:46,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:46,363 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,363 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:46,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:46,363 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:46,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:46,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:46,364 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:46,364 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:46,364 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:46,364 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:46,364 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:46,364 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:46,364 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:46,364 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:46,364 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:46,365 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. started... 2023-03-31 16:55:46,365 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. as already flushing 2023-03-31 16:55:46,365 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:46,365 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:46,365 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:46,365 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,365 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:46,365 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:46,367 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,367 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:46,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:46,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:46,368 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:46,368 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:46,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:46,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:46,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:46,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:46,369 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:46,369 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4733a26[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:46,369 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:46,369 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:46,370 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:46,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:46,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:46,370 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:46,370 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:46,370 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:46,370 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,371 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:46,372 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,372 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:46,372 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:46,372 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:46,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:46,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:46,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:46,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:46,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:46,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:46,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:46,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:46,374 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:46,374 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:46,374 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:46,374 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:46,374 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:46,374 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:46,376 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:46,376 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:46,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:46,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:46,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:46,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:46,376 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:46,377 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:46,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:46,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:46,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:46,377 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:46,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:46,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:46,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:46,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,382 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,382 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:46,382 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:46,382 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,382 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:46,382 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:46,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:46,382 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:46,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:46,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:46,382 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:46,382 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:46,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:46,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:46,382 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:46,382 DEBUG [Thread-3278] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:46,382 DEBUG [Thread-3278] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:46,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:46,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:46,383 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:46,483 DEBUG [Thread-3278] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:46,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:46,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:46,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:46,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:46,486 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:46,486 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:46,486 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:46,486 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:46,486 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:46,486 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:46,487 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,487 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:46,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:46,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:46,488 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,488 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:46,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:46,488 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:46,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:46,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:46,488 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:46,489 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:46,489 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:46,489 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:46,489 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:46,489 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:46,489 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:46,489 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:46,489 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:46,489 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. started... 2023-03-31 16:55:46,489 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. as already flushing 2023-03-31 16:55:46,489 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:46,489 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:46,489 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:46,489 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,489 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:46,489 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:46,496 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:46,496 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:46,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:46,497 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:46,497 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:46,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:46,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:46,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:46,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:46,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:46,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c0be0fd[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:46,498 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:46,498 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:46,501 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:46,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:46,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:46,501 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:46,501 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:46,501 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:46,501 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,501 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:46,502 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:46,502 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,502 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:46,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:46,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:46,502 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:46,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:46,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:46,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:46,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:46,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:46,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:46,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:46,505 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:46,505 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:46,505 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:46,505 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:46,505 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:46,506 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:46,506 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:46,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:46,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:46,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:46,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:46,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:46,506 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:46,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:46,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:46,507 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:46,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:46,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:46,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:46,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:46,508 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:46,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:46,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,510 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,510 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:46,510 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:46,510 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:46,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:46,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:46,510 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:46,510 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:46,510 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:46,510 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:46,510 DEBUG [Thread-3278] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:46,511 DEBUG [Thread-3278] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:46,510 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:46,511 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,511 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:46,511 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:46,511 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:46,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:46,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:46,611 DEBUG [Thread-3278] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:46,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:46,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:46,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:46,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:46,614 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:46,614 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:46,614 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:46,614 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:46,614 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:46,615 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:46,619 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,619 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:46,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:46,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:46,619 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,619 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:46,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:46,620 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:46,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:46,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:46,620 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:46,620 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:46,620 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:46,621 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:46,621 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:46,621 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:46,621 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:46,621 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:46,621 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:46,622 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. started... 2023-03-31 16:55:46,622 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. as already flushing 2023-03-31 16:55:46,622 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:46,622 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:46,622 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:46,622 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,622 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:46,622 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:46,625 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:46,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:46,625 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:46,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:46,626 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:46,626 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:46,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:46,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:46,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:46,628 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:46,628 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3fd06db4[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:46,628 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:46,628 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:46,629 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:46,629 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:46,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:46,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:46,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35494 deadline: 1680281806624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,629 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:46,630 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:46,630 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:46,630 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:46,629 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,630 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:46,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35582 deadline: 1680281806624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,630 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:46,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35546 deadline: 1680281806625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,631 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:46,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:46,631 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:46,631 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:46,631 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:46,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:46,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:46,632 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:46,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35580 deadline: 1680281806625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:46,632 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:46,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35496 deadline: 1680281806625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:46,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:46,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:46,635 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:46,635 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:46,635 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:46,635 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:46,635 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:46,635 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:46,638 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:46,638 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:46,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:46,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:46,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:46,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:46,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:46,638 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:46,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:46,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:46,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:46,638 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:46,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:46,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:46,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:46,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,642 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:46,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:46,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,644 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:46,644 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:46,644 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:46,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:46,644 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:46,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:46,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:46,644 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:46,644 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:46,644 DEBUG [Thread-3278] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:46,644 DEBUG [Thread-3278] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:46,644 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,644 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:46,644 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:46,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:46,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:46,645 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:46,744 DEBUG [Thread-3278] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:46,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:46,747 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:46,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:46,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:46,748 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:46,748 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:46,748 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:46,748 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:46,748 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:46,748 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:46,749 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:46,749 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:46,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:46,750 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,750 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:46,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:46,750 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:46,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:46,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:46,750 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:46,751 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:46,751 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:46,751 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:46,751 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:46,751 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:46,751 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:46,751 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:46,751 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:46,751 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. started... 2023-03-31 16:55:46,751 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. as already flushing 2023-03-31 16:55:46,751 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:46,751 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:46,751 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:46,751 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,751 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:46,751 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:46,753 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,753 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:46,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:46,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:46,753 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:46,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:46,753 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:46,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:46,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:46,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:46,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:46,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46103f3d[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:46,754 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:46,754 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:46,756 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=588 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/B/07ac830d72144b50ac986ad6ca9c61ee 2023-03-31 16:55:46,757 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:46,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:46,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:46,757 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:46,757 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:46,757 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:46,757 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,757 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:46,758 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:46,758 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:46,758 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,758 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:46,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:46,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:46,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:46,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:46,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:46,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:46,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:46,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:46,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:46,761 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:46,761 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:46,761 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:46,761 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:46,761 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:46,763 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:46,763 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:46,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:46,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:46,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:46,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:46,763 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:46,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:46,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:46,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:46,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:46,764 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:46,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:46,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:46,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:46,765 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:46,766 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=588 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/C/7b2b2909a0a34c9097176986b337ba82 2023-03-31 16:55:46,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:46,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,767 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:46,767 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:46,767 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:46,767 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:46,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:46,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:46,768 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:46,767 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:46,768 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:46,768 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,768 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:46,768 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:46,768 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:46,768 DEBUG [Thread-3278] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:46,768 DEBUG [Thread-3278] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:46,768 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:46,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:46,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:46,769 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/A/1af39476ae1e488794a4a84669d8b770 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/1af39476ae1e488794a4a84669d8b770 2023-03-31 16:55:46,773 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/1af39476ae1e488794a4a84669d8b770, entries=200, sequenceid=588, filesize=14.3 K 2023-03-31 16:55:46,773 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/B/07ac830d72144b50ac986ad6ca9c61ee as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/07ac830d72144b50ac986ad6ca9c61ee 2023-03-31 16:55:46,776 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/07ac830d72144b50ac986ad6ca9c61ee, entries=150, sequenceid=588, filesize=11.9 K 2023-03-31 16:55:46,777 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/C/7b2b2909a0a34c9097176986b337ba82 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/7b2b2909a0a34c9097176986b337ba82 2023-03-31 16:55:46,779 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/7b2b2909a0a34c9097176986b337ba82, entries=150, sequenceid=588, filesize=11.9 K 2023-03-31 16:55:46,780 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for fd96548248d41c209e12fca5d7040af5 in 843ms, sequenceid=588, compaction requested=false 2023-03-31 16:55:46,780 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for fd96548248d41c209e12fca5d7040af5: 2023-03-31 16:55:46,868 DEBUG [Thread-3278] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:46,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:46,870 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:46,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:46,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:46,871 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:46,871 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:46,872 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:46,872 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:46,872 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:46,872 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:46,876 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:46,876 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:46,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:46,876 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:46,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:46,876 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:46,877 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:46,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:46,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:46,877 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:46,877 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:46,877 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:46,877 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:46,877 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:46,877 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:46,878 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:46,878 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:46,878 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:46,878 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. started... 2023-03-31 16:55:46,878 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fd96548248d41c209e12fca5d7040af5 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-31 16:55:46,878 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=A 2023-03-31 16:55:46,878 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:46,878 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=B 2023-03-31 16:55:46,878 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:46,878 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=C 2023-03-31 16:55:46,878 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:47,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on fd96548248d41c209e12fca5d7040af5 2023-03-31 16:55:47,134 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. as already flushing 2023-03-31 16:55:47,149 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:47,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35496 deadline: 1680281807142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:47,150 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:47,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35582 deadline: 1680281807148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:47,159 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:47,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35494 deadline: 1680281807149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:47,159 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:47,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35546 deadline: 1680281807149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:47,159 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:47,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35580 deadline: 1680281807150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:47,213 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-31 16:55:47,256 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:47,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35496 deadline: 1680281807251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:47,263 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:47,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35582 deadline: 1680281807258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:47,268 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:47,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35494 deadline: 1680281807260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:47,268 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:47,268 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:47,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35546 deadline: 1680281807260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:47,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35580 deadline: 1680281807261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:47,288 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=609 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/A/113379a1323548739914b0e988e3ab46 2023-03-31 16:55:47,461 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:47,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35496 deadline: 1680281807458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:47,467 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:47,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35582 deadline: 1680281807464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:47,474 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:47,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35546 deadline: 1680281807469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:47,474 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:47,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35494 deadline: 1680281807470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:47,474 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:47,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35580 deadline: 1680281807471, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:47,696 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=609 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/B/45bd50b8c4534d3eac4b65d71bfb3d38 2023-03-31 16:55:47,768 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:47,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35496 deadline: 1680281807764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:47,772 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:47,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35582 deadline: 1680281807769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:47,778 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:47,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35546 deadline: 1680281807775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:47,778 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:47,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35494 deadline: 1680281807775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:47,781 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:47,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35580 deadline: 1680281807776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:48,110 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=609 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/C/ec7afcab177f478c958452d0aff83946 2023-03-31 16:55:48,113 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/A/113379a1323548739914b0e988e3ab46 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/113379a1323548739914b0e988e3ab46 2023-03-31 16:55:48,117 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/113379a1323548739914b0e988e3ab46, entries=150, sequenceid=609, filesize=11.9 K 2023-03-31 16:55:48,117 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/B/45bd50b8c4534d3eac4b65d71bfb3d38 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/45bd50b8c4534d3eac4b65d71bfb3d38 2023-03-31 16:55:48,121 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/45bd50b8c4534d3eac4b65d71bfb3d38, entries=150, sequenceid=609, filesize=11.9 K 2023-03-31 16:55:48,122 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/C/ec7afcab177f478c958452d0aff83946 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/ec7afcab177f478c958452d0aff83946 2023-03-31 16:55:48,126 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/ec7afcab177f478c958452d0aff83946, entries=150, sequenceid=609, filesize=11.9 K 2023-03-31 16:55:48,126 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=73.80 KB/75570 for fd96548248d41c209e12fca5d7040af5 in 1248ms, sequenceid=609, compaction requested=true 2023-03-31 16:55:48,126 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fd96548248d41c209e12fca5d7040af5: 2023-03-31 16:55:48,126 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:48,126 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:48,126 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:48,126 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:48,126 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:48,126 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:48,128 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:48,128 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:48,128 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:48,128 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:48,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:48,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:48,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:48,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:48,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:48,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:48,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:48,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:48,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:48,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5baf8fd3[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:48,130 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:48,131 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:48,132 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:48,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:48,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:48,132 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:48,132 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:48,132 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:48,132 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:48,132 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:48,139 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:48,139 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:48,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:48,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:48,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:48,139 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:48,139 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:48,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:48,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:48,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:48,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:48,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:48,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:48,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:48,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:48,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:48,141 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:48,142 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:48,142 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:48,142 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:48,142 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:48,144 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:48,144 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:48,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:48,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:48,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:48,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:48,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:48,144 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:48,145 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:48,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:48,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:48,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:48,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:48,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:48,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:48,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:48,146 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:48,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:48,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:48,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:48,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:48,149 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:48,149 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:48,149 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:48,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:48,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:48,149 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:48,149 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:48,149 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:48,149 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:48,149 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:48,149 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:48,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:48,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:48,149 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:48,149 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:48,149 DEBUG [Thread-3278] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:48,149 DEBUG [Thread-3278] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:48,149 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:48,150 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:48,250 DEBUG [Thread-3278] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:48,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:48,252 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:48,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:48,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:48,253 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:48,253 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:48,253 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:48,253 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:48,253 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:48,253 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:48,254 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:48,254 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:48,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:48,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:48,255 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:48,255 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:48,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:48,255 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:48,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:48,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:48,256 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:48,256 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:48,256 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:48,256 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:48,256 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:48,256 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:48,256 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:48,256 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:48,257 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:48,257 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. started... 2023-03-31 16:55:48,257 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fd96548248d41c209e12fca5d7040af5 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-31 16:55:48,257 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=A 2023-03-31 16:55:48,257 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:48,257 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=B 2023-03-31 16:55:48,257 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:48,257 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=C 2023-03-31 16:55:48,257 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:48,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on fd96548248d41c209e12fca5d7040af5 2023-03-31 16:55:48,275 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. as already flushing 2023-03-31 16:55:48,301 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:48,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35580 deadline: 1680281808291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:48,302 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:48,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35494 deadline: 1680281808292, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:48,302 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:48,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35546 deadline: 1680281808294, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:48,303 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:48,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35582 deadline: 1680281808295, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:48,303 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:48,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35496 deadline: 1680281808301, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:48,406 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:48,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35580 deadline: 1680281808402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:48,406 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:48,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35494 deadline: 1680281808403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:48,406 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:48,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35546 deadline: 1680281808403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:48,407 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:48,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35582 deadline: 1680281808403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:48,407 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:48,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35496 deadline: 1680281808404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:48,612 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:48,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35496 deadline: 1680281808608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:48,612 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:48,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35580 deadline: 1680281808608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:48,613 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:48,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35494 deadline: 1680281808608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:48,613 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:48,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35546 deadline: 1680281808608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:48,613 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:48,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35582 deadline: 1680281808608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:48,664 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=623 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/A/333a6544de9041a9a9272fbdb4a22627 2023-03-31 16:55:48,919 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:48,919 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:48,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35580 deadline: 1680281808913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:48,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35546 deadline: 1680281808914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:48,919 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:48,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35582 deadline: 1680281808914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:48,919 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:48,920 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:48,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35496 deadline: 1680281808914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:48,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35494 deadline: 1680281808914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,074 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=623 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/B/2bdb0f8176e94f3880f8e6441adcc336 2023-03-31 16:55:49,104 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=623 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/C/5ed60d32bc9c40a59ce00c804b0f3f90 2023-03-31 16:55:49,108 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/A/333a6544de9041a9a9272fbdb4a22627 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/333a6544de9041a9a9272fbdb4a22627 2023-03-31 16:55:49,115 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/333a6544de9041a9a9272fbdb4a22627, entries=150, sequenceid=623, filesize=11.9 K 2023-03-31 16:55:49,116 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/B/2bdb0f8176e94f3880f8e6441adcc336 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/2bdb0f8176e94f3880f8e6441adcc336 2023-03-31 16:55:49,128 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/2bdb0f8176e94f3880f8e6441adcc336, entries=150, sequenceid=623, filesize=11.9 K 2023-03-31 16:55:49,129 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/C/5ed60d32bc9c40a59ce00c804b0f3f90 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/5ed60d32bc9c40a59ce00c804b0f3f90 2023-03-31 16:55:49,133 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/5ed60d32bc9c40a59ce00c804b0f3f90, entries=150, sequenceid=623, filesize=11.9 K 2023-03-31 16:55:49,134 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for fd96548248d41c209e12fca5d7040af5 in 877ms, sequenceid=623, compaction requested=true 2023-03-31 16:55:49,134 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fd96548248d41c209e12fca5d7040af5: 2023-03-31 16:55:49,134 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:49,134 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:49,134 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:49,134 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,134 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:49,134 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:49,136 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,136 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:49,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:49,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:49,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:49,137 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:49,137 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:49,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:49,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:49,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:49,138 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:49,138 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18609053[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:49,138 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:49,138 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:49,140 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:49,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:49,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:49,140 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:49,141 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:49,141 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:49,142 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,142 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:49,142 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:49,142 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:49,143 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:49,143 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:49,143 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:49,143 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:49,143 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:49,143 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:49,143 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:49,145 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:49,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:49,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:49,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:49,145 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:49,145 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:49,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:49,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:49,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:49,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:49,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:49,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:49,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:49,146 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:49,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:49,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:49,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:49,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,150 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,152 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,152 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:49,152 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:49,152 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:49,152 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,152 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:49,152 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:49,152 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:49,152 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:49,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:49,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:49,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:49,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:49,152 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:49,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:49,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:49,153 DEBUG [Thread-3278] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:49,153 DEBUG [Thread-3278] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:49,253 DEBUG [Thread-3278] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:49,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:49,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:49,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:49,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:49,256 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:49,256 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:49,256 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:49,256 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:49,257 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:49,257 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:49,259 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:49,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:49,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:49,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:49,260 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,260 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:49,261 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,261 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:49,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:49,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:49,261 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:49,261 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:49,261 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:49,261 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:49,261 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:49,261 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:49,263 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:49,263 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:49,263 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:49,263 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. started... 2023-03-31 16:55:49,263 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fd96548248d41c209e12fca5d7040af5 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-31 16:55:49,263 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=A 2023-03-31 16:55:49,263 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:49,263 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=B 2023-03-31 16:55:49,263 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:49,263 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=C 2023-03-31 16:55:49,264 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:49,286 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=641 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/A/733dbb73607a4da588aaff5ce014c34c 2023-03-31 16:55:49,305 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=641 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/B/9146e335fac246dc94fdd2a6c259335c 2023-03-31 16:55:49,326 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=641 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/C/90253ba9fc1b4b069c9741467cdaf888 2023-03-31 16:55:49,331 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/A/733dbb73607a4da588aaff5ce014c34c as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/733dbb73607a4da588aaff5ce014c34c 2023-03-31 16:55:49,337 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/733dbb73607a4da588aaff5ce014c34c, entries=150, sequenceid=641, filesize=11.9 K 2023-03-31 16:55:49,338 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/B/9146e335fac246dc94fdd2a6c259335c as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/9146e335fac246dc94fdd2a6c259335c 2023-03-31 16:55:49,342 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/9146e335fac246dc94fdd2a6c259335c, entries=150, sequenceid=641, filesize=11.9 K 2023-03-31 16:55:49,343 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/C/90253ba9fc1b4b069c9741467cdaf888 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/90253ba9fc1b4b069c9741467cdaf888 2023-03-31 16:55:49,349 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/90253ba9fc1b4b069c9741467cdaf888, entries=150, sequenceid=641, filesize=11.9 K 2023-03-31 16:55:49,350 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=0 B/0 for fd96548248d41c209e12fca5d7040af5 in 87ms, sequenceid=641, compaction requested=true 2023-03-31 16:55:49,350 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fd96548248d41c209e12fca5d7040af5: 2023-03-31 16:55:49,350 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:49,350 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:49,350 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:49,350 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,350 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:49,350 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:49,352 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:49,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:49,352 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:49,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:49,353 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:49,353 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:49,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:49,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:49,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:49,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:49,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24e1e326[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:49,354 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:49,354 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:49,356 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:49,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:49,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:49,357 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:49,357 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:49,357 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:49,357 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,357 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:49,359 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:49,359 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:49,359 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:49,359 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:49,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:49,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:49,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:49,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:49,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:49,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:49,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:49,361 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:49,362 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:49,362 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:49,362 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:49,362 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:49,362 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:49,365 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:49,365 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:49,365 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:49,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:49,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:49,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:49,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:49,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:49,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:49,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:49,366 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:49,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:49,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:49,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:49,369 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:49,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:49,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:49,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,372 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:49,372 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:49,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:49,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:49,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:49,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:49,380 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,380 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:49,380 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:49,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:49,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:49,380 DEBUG [Thread-3278] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:49,380 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:49,380 DEBUG [Thread-3278] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:49,380 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,381 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:49,381 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:49,381 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:49,437 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing fd96548248d41c209e12fca5d7040af5 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-31 16:55:49,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on fd96548248d41c209e12fca5d7040af5 2023-03-31 16:55:49,437 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=A 2023-03-31 16:55:49,438 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:49,438 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=B 2023-03-31 16:55:49,438 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:49,438 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=C 2023-03-31 16:55:49,438 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:49,480 DEBUG [Thread-3278] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:49,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:49,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:49,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:49,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:49,484 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:49,484 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:49,484 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:49,484 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:49,484 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:49,484 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:49,486 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,486 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:49,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:49,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:49,486 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,486 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:49,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:49,486 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:49,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:49,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:49,487 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:49,487 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:49,487 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:49,487 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:49,487 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:49,487 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:49,488 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:49,488 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:49,488 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:49,488 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. started... 2023-03-31 16:55:49,488 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. as already flushing 2023-03-31 16:55:49,488 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:49,488 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:49,488 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:49,488 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,488 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:49,488 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:49,489 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:49,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35496 deadline: 1680281809475, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,489 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:49,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35494 deadline: 1680281809476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,489 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680281749463, max timestamp=1680281749487Num uniques 859; 2023-03-31 16:55:49,489 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-31 16:55:49,490 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:49,490 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:49,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:49,490 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:49,490 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:49,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:49,491 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680281749463, max timestamp=1680281749487Num uniques 859; 2023-03-31 16:55:49,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:49,491 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-31 16:55:49,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:49,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:49,492 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:49,492 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680281749463, max timestamp=1680281749487Num uniques 859; 2023-03-31 16:55:49,492 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:49,492 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-31 16:55:49,492 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:49,492 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@634fd950[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:49,494 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:49,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:49,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:49,494 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:49,494 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:49,494 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:49,494 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,494 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:49,496 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,496 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:49,496 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:49,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:49,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:49,496 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:49,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:49,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:49,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:49,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:49,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:49,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:49,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:49,498 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:49,498 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:49,498 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:49,498 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:49,498 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:49,502 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:49,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:49,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:49,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:49,503 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:49,503 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:49,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:49,503 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:49,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:49,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:49,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:49,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:49,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:49,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:49,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:49,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,505 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:49,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:49,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,507 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,507 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:49,507 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:49,507 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:49,507 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:49,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:49,507 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:49,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:49,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:49,508 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:49,507 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:49,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:49,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:49,508 DEBUG [Thread-3278] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:49,508 DEBUG [Thread-3278] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:49,508 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:49,508 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:49,544 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:49,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35546 deadline: 1680281809531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,544 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:49,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35580 deadline: 1680281809532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,551 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:49,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35582 deadline: 1680281809542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,597 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:49,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35496 deadline: 1680281809590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,597 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:49,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35494 deadline: 1680281809590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,608 DEBUG [Thread-3278] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:49,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:49,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:49,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:49,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:49,611 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:49,611 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:49,611 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:49,611 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:49,611 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:49,612 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:49,613 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:49,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:49,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:49,613 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:49,614 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,614 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:49,614 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:49,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:49,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:49,614 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:49,619 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:49,619 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:49,619 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:49,619 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:49,619 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:49,619 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:49,619 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:49,619 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:49,619 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. started... 2023-03-31 16:55:49,620 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. as already flushing 2023-03-31 16:55:49,620 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:49,620 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:49,620 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:49,620 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,620 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:49,620 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:49,623 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,623 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:49,623 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:49,623 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:49,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:49,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:49,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:49,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:49,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:49,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:49,625 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:49,625 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@443913fd[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:49,625 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:49,625 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:49,626 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:49,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:49,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:49,627 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:49,627 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,627 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:49,627 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:49,627 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:49,629 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:49,629 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:49,629 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,629 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:49,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:49,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:49,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:49,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:49,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:49,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:49,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:49,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:49,631 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:49,631 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:49,631 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:49,631 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:49,631 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:49,631 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:49,633 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:49,633 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:49,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:49,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:49,633 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:49,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:49,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:49,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:49,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:49,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:49,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:49,633 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:49,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:49,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:49,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:49,634 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:49,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:49,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,637 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,637 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:49,637 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:49,637 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,637 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:49,637 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:49,637 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:49,637 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:49,637 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:49,637 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:49,638 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:49,637 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:49,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:49,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:49,638 DEBUG [Thread-3278] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:49,638 DEBUG [Thread-3278] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:49,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:49,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:49,638 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:49,652 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:49,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35580 deadline: 1680281809645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,652 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:49,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35546 deadline: 1680281809645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,661 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:49,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35582 deadline: 1680281809652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,738 DEBUG [Thread-3278] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:49,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:49,740 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:49,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:49,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:49,742 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:49,742 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:49,742 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:49,742 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:49,743 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:49,743 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:49,745 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:49,745 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:49,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:49,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:49,745 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,745 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:49,745 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:49,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:49,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:49,746 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:49,746 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:49,746 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:49,746 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:49,746 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:49,746 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:49,746 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:49,746 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:49,747 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:49,747 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. started... 2023-03-31 16:55:49,747 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. as already flushing 2023-03-31 16:55:49,747 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:49,747 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:49,747 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:49,747 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,747 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:49,747 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:49,748 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,748 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:49,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:49,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:49,749 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:49,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:49,749 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:49,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:49,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:49,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:49,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:49,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58c7e121[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:49,752 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:49,752 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:49,753 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:49,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:49,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:49,754 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:49,754 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:49,754 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:49,755 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,755 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:49,756 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,756 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:49,756 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:49,757 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:49,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:49,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:49,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:49,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:49,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:49,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:49,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:49,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:49,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:49,759 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:49,759 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:49,759 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:49,759 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:49,759 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:49,761 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:49,761 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:49,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:49,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:49,761 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:49,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:49,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:49,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:49,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:49,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:49,762 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:49,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:49,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:49,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:49,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:49,764 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:49,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:49,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,767 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:49,767 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:49,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:49,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:49,767 DEBUG [Thread-3278] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:49,767 DEBUG [Thread-3278] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:49,768 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:49,768 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:49,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:49,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:49,768 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,768 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:49,768 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:49,768 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,768 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:49,768 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:49,768 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:49,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:49,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:49,810 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:49,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35496 deadline: 1680281809798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,811 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:49,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35494 deadline: 1680281809799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,854 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=652 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/A/13818983462f4ee9a100843f5c7e7a63 2023-03-31 16:55:49,865 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:49,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35546 deadline: 1680281809854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,867 DEBUG [Thread-3278] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:49,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:49,869 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:49,870 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:49,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35580 deadline: 1680281809856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,870 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:49,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35582 deadline: 1680281809864, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:49,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:49,871 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:49,871 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:49,871 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:49,871 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:49,871 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:49,871 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:49,872 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,872 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:49,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:49,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:49,873 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,873 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:49,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:49,873 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:49,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:49,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:49,874 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:49,874 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:49,874 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:49,874 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:49,874 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:49,874 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:49,880 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:49,880 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:49,880 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:49,880 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. started... 2023-03-31 16:55:49,880 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. as already flushing 2023-03-31 16:55:49,880 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:49,880 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:49,880 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:49,880 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,880 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:49,880 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:49,883 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,883 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:49,883 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:49,883 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:49,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:49,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:49,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:49,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:49,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:49,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:49,885 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:49,885 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7314d1cc[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:49,885 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:49,885 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:49,886 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:49,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:49,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:49,886 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:49,886 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:49,886 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:49,887 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,887 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:49,888 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:49,888 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:49,888 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,889 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:49,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:49,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:49,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:49,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:49,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:49,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:49,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:49,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:49,891 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:49,891 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:49,891 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:49,891 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:49,891 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:49,891 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:49,893 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:49,893 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:49,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:49,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:49,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:49,893 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:49,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:49,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:49,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:49,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:49,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:49,894 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:49,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:49,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:49,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:49,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,895 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:49,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:49,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,901 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,901 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:49,901 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:49,901 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:49,901 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:49,901 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:49,902 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:49,902 DEBUG [Thread-3278] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:49,901 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:49,901 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:49,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:49,902 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:49,902 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:49,902 DEBUG [Thread-3278] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:49,902 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:49,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:49,902 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:49,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:49,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:50,002 DEBUG [Thread-3278] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:50,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:50,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:50,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:50,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:50,005 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:50,005 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:50,005 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:50,005 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:50,006 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:50,006 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:50,007 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,007 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:50,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:50,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:50,008 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,008 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:50,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:50,008 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:50,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:50,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:50,009 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:50,009 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:50,009 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:50,009 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:50,009 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:50,009 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:50,016 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:50,016 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:50,016 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:50,016 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. started... 2023-03-31 16:55:50,016 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. as already flushing 2023-03-31 16:55:50,016 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:50,016 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:50,016 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:50,016 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,016 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:50,016 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:50,018 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:50,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:50,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:50,019 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:50,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:50,019 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:50,019 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:50,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:50,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:50,020 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:50,020 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e8c13a7[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:50,020 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:50,020 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:50,022 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:50,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:50,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:50,022 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:50,022 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:50,022 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:50,023 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,023 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:50,024 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:50,024 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:50,024 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,024 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:50,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:50,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:50,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:50,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:50,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:50,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:50,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:50,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:50,026 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:50,026 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:50,026 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:50,026 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:50,026 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:50,026 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:50,028 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:50,028 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:50,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:50,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:50,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:50,028 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:50,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:50,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:50,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:50,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:50,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:50,028 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:50,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:50,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:50,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:50,029 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:50,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:50,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,031 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,031 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:50,031 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:50,031 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:50,032 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:50,031 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:50,031 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:50,032 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:50,032 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,032 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:50,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:50,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:50,032 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:50,032 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:50,032 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:50,032 DEBUG [Thread-3278] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:50,032 DEBUG [Thread-3278] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:50,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:50,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:50,117 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:50,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35496 deadline: 1680281810112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,123 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:50,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35494 deadline: 1680281810114, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,132 DEBUG [Thread-3278] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:50,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:50,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:50,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:50,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:50,136 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:50,136 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:50,136 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:50,136 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:50,136 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:50,136 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:50,143 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:50,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:50,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:50,143 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:50,144 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,144 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:50,144 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:50,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:50,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:50,145 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:50,145 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:50,145 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:50,145 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:50,145 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:50,145 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:50,148 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:50,148 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:50,148 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:50,148 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. started... 2023-03-31 16:55:50,148 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. as already flushing 2023-03-31 16:55:50,148 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:50,148 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:50,148 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:50,148 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,148 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:50,149 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:50,150 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:50,150 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:50,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:50,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:50,153 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:50,153 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:50,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:50,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:50,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:50,154 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:50,154 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62265370[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:50,154 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:50,154 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:50,161 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:50,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:50,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:50,161 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:50,161 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:50,161 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:50,161 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,162 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:50,163 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,163 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:50,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,163 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:50,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:50,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:50,163 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:50,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:50,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:50,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:50,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:50,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:50,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:50,167 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:50,167 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:50,167 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:50,167 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:50,167 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:50,167 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:50,170 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:50,170 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:50,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:50,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:50,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:50,170 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:50,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:50,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:50,170 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:50,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:50,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:50,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:50,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:50,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:50,172 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:50,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:50,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:50,174 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,174 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:50,174 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:50,174 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,174 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:50,174 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:50,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:50,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:50,175 DEBUG [Thread-3278] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:50,175 DEBUG [Thread-3278] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:50,174 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:50,174 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:50,175 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:50,175 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:50,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:50,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:50,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:50,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:50,175 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:50,182 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:50,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35546 deadline: 1680281810169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,182 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:50,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35580 deadline: 1680281810171, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,182 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:50,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35582 deadline: 1680281810174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,264 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=652 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/B/c6d85b68e0024b34bc43362dd074eb22 2023-03-31 16:55:50,275 DEBUG [Thread-3278] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:50,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:50,277 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:50,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:50,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:50,278 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:50,278 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:50,278 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:50,278 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:50,279 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:50,279 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:50,280 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:50,280 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:50,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:50,280 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,280 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:50,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:50,281 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:50,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:50,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:50,281 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:50,282 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:50,282 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:50,282 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:50,282 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:50,282 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:50,282 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:50,282 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:50,282 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:50,282 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. started... 2023-03-31 16:55:50,282 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. as already flushing 2023-03-31 16:55:50,282 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:50,282 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:50,282 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:50,282 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,283 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:50,283 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:50,284 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,284 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:50,285 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:50,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:50,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:50,285 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:50,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:50,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:50,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:50,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:50,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:50,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50f68fc[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:50,287 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:50,287 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:50,289 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:50,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:50,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:50,290 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:50,290 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:50,290 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:50,290 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,290 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:50,291 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,291 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:50,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:50,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:50,291 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:50,291 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:50,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:50,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:50,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:50,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:50,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:50,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:50,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:50,294 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:50,294 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:50,294 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:50,294 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:50,294 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:50,295 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:50,295 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:50,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:50,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:50,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:50,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:50,295 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:50,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:50,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:50,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:50,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:50,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:50,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:50,297 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:50,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:50,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,300 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:50,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:50,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,306 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,306 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:50,306 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:50,306 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,306 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:50,306 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:50,306 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:50,306 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:50,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:50,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:50,307 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:50,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:50,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:50,307 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:50,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:50,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:50,307 DEBUG [Thread-3278] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:50,307 DEBUG [Thread-3278] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:50,407 DEBUG [Thread-3278] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:50,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:50,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:50,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:50,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:50,411 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:50,411 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:50,411 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:50,411 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:50,411 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:50,411 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:50,412 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,412 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:50,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:50,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:50,413 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,413 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:50,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:50,413 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:50,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:50,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:50,414 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:50,414 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:50,414 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:50,414 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:50,414 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:50,414 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:50,414 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:50,414 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:50,414 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:50,414 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. started... 2023-03-31 16:55:50,414 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. as already flushing 2023-03-31 16:55:50,414 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:50,415 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:50,415 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:50,415 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,415 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:50,415 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:50,416 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,416 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:50,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:50,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:50,417 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:50,417 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:50,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:50,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:50,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:50,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:50,419 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:50,419 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50c04320[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:50,419 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:50,419 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:50,421 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:50,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:50,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:50,422 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:50,422 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:50,422 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:50,422 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,422 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:50,423 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:50,423 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:50,423 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,423 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:50,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:50,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:50,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:50,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:50,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:50,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:50,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:50,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:50,426 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:50,426 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:50,426 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:50,426 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:50,426 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:50,426 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:50,427 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:50,427 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:50,428 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:50,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:50,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:50,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:50,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:50,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:50,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:50,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:50,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:50,430 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:50,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:50,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:50,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:50,432 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:50,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:50,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,434 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,434 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:50,434 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:50,434 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:50,435 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:50,435 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:50,435 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:50,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:50,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:50,435 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:50,435 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:50,435 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:50,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:50,435 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:50,435 DEBUG [Thread-3278] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:50,435 DEBUG [Thread-3278] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:50,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:50,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:50,535 DEBUG [Thread-3278] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:50,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:50,537 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:50,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:50,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:50,538 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:50,538 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:50,538 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:50,538 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:50,539 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:50,539 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:50,541 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:50,541 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:50,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:50,541 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,541 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:50,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:50,541 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:50,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:50,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:50,542 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:50,542 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:50,542 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:50,542 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:50,542 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:50,542 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:50,542 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:50,542 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:50,542 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:50,542 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. started... 2023-03-31 16:55:50,543 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. as already flushing 2023-03-31 16:55:50,543 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:50,543 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:50,543 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:50,543 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,543 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:50,543 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:50,545 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:50,545 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:50,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:50,545 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:50,545 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:50,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:50,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:50,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:50,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:50,546 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:50,546 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e5f73b4[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:50,546 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:50,546 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:50,547 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:50,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:50,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:50,548 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:50,548 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:50,548 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:50,549 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,549 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:50,549 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:50,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:50,550 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:50,550 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:50,550 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:50,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:50,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:50,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:50,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:50,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:50,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:50,552 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:50,552 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:50,552 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:50,552 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:50,552 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:50,552 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:50,558 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:50,558 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:50,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:50,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:50,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:50,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:50,558 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:50,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:50,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:50,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:50,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:50,559 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:50,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:50,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:50,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:50,560 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:50,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:50,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,562 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,562 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:50,562 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:50,562 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,562 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:50,562 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:50,562 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:50,562 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:50,562 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:50,562 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:50,562 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:50,562 DEBUG [Thread-3278] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:50,562 DEBUG [Thread-3278] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:50,562 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:50,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:50,562 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:50,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:50,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:50,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:50,632 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:50,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35496 deadline: 1680281810625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,633 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:50,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35494 deadline: 1680281810626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,662 DEBUG [Thread-3278] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:50,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:50,665 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:50,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:50,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:50,666 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:50,666 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:50,666 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:50,666 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:50,666 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:50,666 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:50,668 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,668 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:50,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:50,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:50,668 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,668 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:50,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:50,668 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:50,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:50,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:50,669 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:50,669 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:50,669 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:50,669 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:50,669 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:50,669 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:50,669 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:50,669 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:50,670 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:50,670 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. started... 2023-03-31 16:55:50,670 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. as already flushing 2023-03-31 16:55:50,670 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:50,670 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:50,670 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:50,670 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,670 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:50,670 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:50,671 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,671 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:50,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:50,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:50,672 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:50,672 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:50,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:50,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:50,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:50,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:50,673 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:50,673 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e3206dc[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:50,673 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:50,673 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:50,675 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:50,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:50,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:50,675 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:50,675 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:50,675 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,675 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:50,675 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:50,677 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=652 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/C/de63658e2b154329b801542b6bd4d830 2023-03-31 16:55:50,677 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,677 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:50,677 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:50,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:50,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:50,677 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:50,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:50,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:50,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:50,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:50,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:50,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:50,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:50,680 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:50,681 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:50,681 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:50,681 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:50,681 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:50,681 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/A/13818983462f4ee9a100843f5c7e7a63 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/13818983462f4ee9a100843f5c7e7a63 2023-03-31 16:55:50,682 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:50,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:50,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:50,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:50,682 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:50,682 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:50,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:50,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:50,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:50,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:50,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:50,683 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:50,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:50,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:50,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:50,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,685 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,685 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/13818983462f4ee9a100843f5c7e7a63, entries=250, sequenceid=652, filesize=16.7 K 2023-03-31 16:55:50,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:50,685 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/B/c6d85b68e0024b34bc43362dd074eb22 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/c6d85b68e0024b34bc43362dd074eb22 2023-03-31 16:55:50,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:50,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,688 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:50,688 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:50,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:50,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:50,688 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,688 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:50,688 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:50,689 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:50,689 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,689 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:50,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:50,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:50,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:50,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:50,689 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:50,689 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:50,689 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:50,689 DEBUG [Thread-3278] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:50,689 DEBUG [Thread-3278] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:50,689 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/c6d85b68e0024b34bc43362dd074eb22, entries=150, sequenceid=652, filesize=11.9 K 2023-03-31 16:55:50,690 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/C/de63658e2b154329b801542b6bd4d830 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/de63658e2b154329b801542b6bd4d830 2023-03-31 16:55:50,690 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:50,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35546 deadline: 1680281810683, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,691 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:50,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35582 deadline: 1680281810685, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,693 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/de63658e2b154329b801542b6bd4d830, entries=150, sequenceid=652, filesize=11.9 K 2023-03-31 16:55:50,694 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for fd96548248d41c209e12fca5d7040af5 in 1257ms, sequenceid=652, compaction requested=true 2023-03-31 16:55:50,694 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for fd96548248d41c209e12fca5d7040af5: 2023-03-31 16:55:50,694 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:55:50,694 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-31 16:55:50,694 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-31 16:55:50,694 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:55:50,694 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:55:50,696 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 80710 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-31 16:55:50,696 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1912): fd96548248d41c209e12fca5d7040af5/A is initiating minor compaction (all files) 2023-03-31 16:55:50,696 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73390 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-31 16:55:50,696 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of fd96548248d41c209e12fca5d7040af5/A in TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:50,696 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): fd96548248d41c209e12fca5d7040af5/B is initiating minor compaction (all files) 2023-03-31 16:55:50,696 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/ac1a7331a0ee480abd994c5979aa85af, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/1af39476ae1e488794a4a84669d8b770, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/113379a1323548739914b0e988e3ab46, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/333a6544de9041a9a9272fbdb4a22627, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/733dbb73607a4da588aaff5ce014c34c, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/13818983462f4ee9a100843f5c7e7a63] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp, totalSize=78.8 K 2023-03-31 16:55:50,696 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fd96548248d41c209e12fca5d7040af5/B in TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:50,696 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/353ad655953f4c29bb1028c2855023ff, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/07ac830d72144b50ac986ad6ca9c61ee, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/45bd50b8c4534d3eac4b65d71bfb3d38, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/2bdb0f8176e94f3880f8e6441adcc336, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/9146e335fac246dc94fdd2a6c259335c, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/c6d85b68e0024b34bc43362dd074eb22] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp, totalSize=71.7 K 2023-03-31 16:55:50,696 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting ac1a7331a0ee480abd994c5979aa85af, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=574, earliestPutTs=1680281745848 2023-03-31 16:55:50,697 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 353ad655953f4c29bb1028c2855023ff, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=574, earliestPutTs=1680281745848 2023-03-31 16:55:50,697 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 1af39476ae1e488794a4a84669d8b770, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=588, earliestPutTs=1680281745871 2023-03-31 16:55:50,697 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 07ac830d72144b50ac986ad6ca9c61ee, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=588, earliestPutTs=1680281745871 2023-03-31 16:55:50,697 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 113379a1323548739914b0e988e3ab46, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=609, earliestPutTs=1680281745957 2023-03-31 16:55:50,697 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 45bd50b8c4534d3eac4b65d71bfb3d38, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=609, earliestPutTs=1680281745957 2023-03-31 16:55:50,697 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 333a6544de9041a9a9272fbdb4a22627, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=623, earliestPutTs=1680281747146 2023-03-31 16:55:50,698 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 2bdb0f8176e94f3880f8e6441adcc336, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=623, earliestPutTs=1680281747146 2023-03-31 16:55:50,698 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 733dbb73607a4da588aaff5ce014c34c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=641, earliestPutTs=1680281748293 2023-03-31 16:55:50,698 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 9146e335fac246dc94fdd2a6c259335c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=641, earliestPutTs=1680281748293 2023-03-31 16:55:50,698 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 13818983462f4ee9a100843f5c7e7a63, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=652, earliestPutTs=1680281749426 2023-03-31 16:55:50,698 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting c6d85b68e0024b34bc43362dd074eb22, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=652, earliestPutTs=1680281749433 2023-03-31 16:55:50,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on fd96548248d41c209e12fca5d7040af5 2023-03-31 16:55:50,699 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing fd96548248d41c209e12fca5d7040af5 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-03-31 16:55:50,700 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=A 2023-03-31 16:55:50,700 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:55:50,700 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=B 2023-03-31 16:55:50,700 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:55:50,700 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=C 2023-03-31 16:55:50,700 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:55:50,714 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): fd96548248d41c209e12fca5d7040af5#B#compaction#471 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:55:50,716 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] throttle.PressureAwareThroughputController(145): fd96548248d41c209e12fca5d7040af5#A#compaction#472 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:55:50,731 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/A/a311336456b9497a86565ffef7c50086 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/a311336456b9497a86565ffef7c50086 2023-03-31 16:55:50,735 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in fd96548248d41c209e12fca5d7040af5/A of fd96548248d41c209e12fca5d7040af5 into a311336456b9497a86565ffef7c50086(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:55:50,735 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for fd96548248d41c209e12fca5d7040af5: 2023-03-31 16:55:50,735 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5., storeName=fd96548248d41c209e12fca5d7040af5/A, priority=10, startTime=1680281750694; duration=0sec 2023-03-31 16:55:50,735 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:55:50,735 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-31 16:55:50,736 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73390 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-31 16:55:50,736 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1912): fd96548248d41c209e12fca5d7040af5/C is initiating minor compaction (all files) 2023-03-31 16:55:50,737 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of fd96548248d41c209e12fca5d7040af5/C in TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:50,737 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/9e45e98286c94a5dbd25d44026aa9e03, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/7b2b2909a0a34c9097176986b337ba82, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/ec7afcab177f478c958452d0aff83946, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/5ed60d32bc9c40a59ce00c804b0f3f90, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/90253ba9fc1b4b069c9741467cdaf888, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/de63658e2b154329b801542b6bd4d830] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp, totalSize=71.7 K 2023-03-31 16:55:50,737 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 9e45e98286c94a5dbd25d44026aa9e03, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=574, earliestPutTs=1680281745848 2023-03-31 16:55:50,737 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 7b2b2909a0a34c9097176986b337ba82, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=588, earliestPutTs=1680281745871 2023-03-31 16:55:50,738 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting ec7afcab177f478c958452d0aff83946, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=609, earliestPutTs=1680281745957 2023-03-31 16:55:50,738 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 5ed60d32bc9c40a59ce00c804b0f3f90, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=623, earliestPutTs=1680281747146 2023-03-31 16:55:50,738 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 90253ba9fc1b4b069c9741467cdaf888, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=641, earliestPutTs=1680281748293 2023-03-31 16:55:50,738 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting de63658e2b154329b801542b6bd4d830, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=652, earliestPutTs=1680281749433 2023-03-31 16:55:50,748 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] throttle.PressureAwareThroughputController(145): fd96548248d41c209e12fca5d7040af5#C#compaction#473 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:55:50,756 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/C/0d1d9f722a6542ee9feaa02e210b8abc as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/0d1d9f722a6542ee9feaa02e210b8abc 2023-03-31 16:55:50,759 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in fd96548248d41c209e12fca5d7040af5/C of fd96548248d41c209e12fca5d7040af5 into 0d1d9f722a6542ee9feaa02e210b8abc(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:55:50,759 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for fd96548248d41c209e12fca5d7040af5: 2023-03-31 16:55:50,759 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5., storeName=fd96548248d41c209e12fca5d7040af5/C, priority=10, startTime=1680281750694; duration=0sec 2023-03-31 16:55:50,760 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:55:50,770 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:50,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35580 deadline: 1680281810769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,789 DEBUG [Thread-3278] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:50,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:50,791 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:50,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:50,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:50,793 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:50,793 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:50,793 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:50,793 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:50,793 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:50,793 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:50,795 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:50,795 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:50,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:50,796 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,796 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:50,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:50,796 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:50,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:50,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:50,796 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:50,798 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:50,798 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:50,798 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:50,798 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:50,798 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:50,798 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:50,798 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:50,798 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:50,799 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. started... 2023-03-31 16:55:50,799 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. as already flushing 2023-03-31 16:55:50,799 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:50,799 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:50,799 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:50,799 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,799 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:50,799 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:50,805 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:50,805 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:50,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:50,806 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:50,806 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:50,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:50,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:50,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:50,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:50,807 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:50,807 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7004c76e[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:50,807 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:50,807 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:50,811 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:50,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:50,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:50,811 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:50,811 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:50,811 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,811 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:50,811 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:50,812 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,812 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:50,812 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:50,812 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:50,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:50,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:50,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:50,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:50,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:50,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:50,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:50,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:50,815 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:50,815 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:50,815 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:50,815 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:50,815 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:50,815 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:50,817 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:50,817 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:50,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:50,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:50,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:50,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:50,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:50,817 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:50,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:50,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:50,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:50,818 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:50,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:50,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:50,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:50,819 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:50,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:50,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,822 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,822 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:50,822 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:50,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:50,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:50,822 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:50,822 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:50,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:50,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:50,822 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:50,822 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:50,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:50,822 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:50,822 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:50,822 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:50,823 DEBUG [Thread-3278] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:50,823 DEBUG [Thread-3278] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:50,875 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:50,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35580 deadline: 1680281810871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,923 DEBUG [Thread-3278] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:50,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:50,925 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:50,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:50,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:50,926 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:50,926 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:50,926 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:50,926 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:50,927 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:50,927 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:50,928 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,928 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:50,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:50,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:50,929 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,929 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:50,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:50,929 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:50,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:50,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:50,929 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:50,929 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:50,929 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:50,930 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:50,930 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:50,930 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:50,930 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:50,930 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:50,930 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:50,930 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. started... 2023-03-31 16:55:50,930 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. as already flushing 2023-03-31 16:55:50,930 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:50,930 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:50,930 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:50,930 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,930 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:50,930 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:50,932 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,932 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:50,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:50,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:50,932 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:50,932 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:50,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:50,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:50,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:50,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:50,933 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:50,933 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@228f530d[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:50,933 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:50,933 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:50,934 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:50,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:50,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:50,935 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:50,935 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:50,935 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:50,935 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,935 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:50,941 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:50,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:50,941 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:50,941 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:50,941 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:50,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:50,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:50,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:50,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:50,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:50,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:50,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:50,943 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:50,943 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:50,943 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:50,943 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:50,943 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:50,945 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:50,945 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:50,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:50,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:50,945 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:50,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:50,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:50,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:50,945 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:50,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:50,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:50,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:50,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:50,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:50,946 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:50,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:50,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:50,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,948 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,949 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:50,949 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:50,949 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:50,949 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:50,949 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:50,949 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:50,949 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:50,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:50,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:50,949 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:50,949 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:50,949 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:50,949 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:50,949 DEBUG [Thread-3278] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:50,949 DEBUG [Thread-3278] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:50,949 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:50,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:50,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:51,049 DEBUG [Thread-3278] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:51,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:51,051 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:51,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:51,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:51,052 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:51,052 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:51,052 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:51,052 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:51,053 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:51,053 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:51,054 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,054 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:51,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:51,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:51,055 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,055 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:51,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:51,055 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:51,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:51,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:51,055 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:51,056 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:51,056 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:51,056 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:51,056 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:51,056 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:51,056 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:51,056 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:51,056 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. started... 2023-03-31 16:55:51,056 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. as already flushing 2023-03-31 16:55:51,056 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:51,056 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:51,056 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:51,056 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,056 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:51,056 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:51,058 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,059 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:51,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:51,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:51,059 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:51,059 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:51,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:51,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:51,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:51,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:51,060 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:51,060 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@768463b[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:51,061 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:51,061 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:51,062 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:51,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:51,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:51,062 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:51,062 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:51,062 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:51,062 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,062 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:51,063 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,063 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:51,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:51,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:51,064 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:51,064 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:51,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:51,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:51,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:51,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:51,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:51,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:51,066 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:51,066 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:51,066 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:51,066 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:51,066 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:51,066 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:51,067 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:51,067 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:51,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:51,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:51,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:51,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:51,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:51,067 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:51,068 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:51,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:51,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:51,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:51,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:51,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:51,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:51,069 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:51,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:51,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,076 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:51,076 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:51,076 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:51,076 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:51,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:51,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:51,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:51,076 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:51,077 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:51,077 DEBUG [Thread-3278] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:51,077 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:51,077 DEBUG [Thread-3278] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:51,077 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,077 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:51,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:51,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:51,077 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:51,077 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:51,083 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:51,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35580 deadline: 1680281811078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,128 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=687 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/A/be75b33618904778a2f9484b9332aade 2023-03-31 16:55:51,131 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/B/a323f04d9f0d484cbe71c78cbecdb1e4 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/a323f04d9f0d484cbe71c78cbecdb1e4 2023-03-31 16:55:51,137 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in fd96548248d41c209e12fca5d7040af5/B of fd96548248d41c209e12fca5d7040af5 into a323f04d9f0d484cbe71c78cbecdb1e4(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:55:51,137 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fd96548248d41c209e12fca5d7040af5: 2023-03-31 16:55:51,137 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5., storeName=fd96548248d41c209e12fca5d7040af5/B, priority=10, startTime=1680281750694; duration=0sec 2023-03-31 16:55:51,137 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:55:51,177 DEBUG [Thread-3278] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:51,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:51,179 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:51,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:51,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:51,181 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:51,181 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:51,181 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:51,181 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:51,182 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:51,182 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:51,183 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,183 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:51,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:51,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:51,183 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,183 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:51,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:51,184 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:51,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:51,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:51,184 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:51,184 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:51,185 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:51,185 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:51,185 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:51,185 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:51,185 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:51,185 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:51,185 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:51,185 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. started... 2023-03-31 16:55:51,185 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. as already flushing 2023-03-31 16:55:51,185 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:51,185 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:51,185 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:51,185 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,185 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:51,185 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:51,187 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,187 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:51,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:51,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:51,187 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:51,187 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:51,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:51,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:51,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:51,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:51,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:51,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19f2a4ec[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:51,188 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:51,189 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:51,191 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:51,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:51,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:51,191 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:51,191 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,191 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:51,191 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:51,191 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:51,193 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,193 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:51,193 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:51,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:51,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:51,193 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:51,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:51,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:51,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:51,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:51,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:51,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:51,195 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:51,195 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:51,195 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:51,195 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:51,195 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:51,195 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:51,197 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:51,197 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:51,197 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:51,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:51,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:51,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:51,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:51,198 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:51,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:51,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:51,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:51,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:51,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:51,199 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:51,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:51,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:51,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:51,201 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,201 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:51,201 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:51,201 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:51,201 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:51,201 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:51,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:51,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:51,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:51,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:51,201 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,201 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:51,201 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:51,201 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:51,201 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:51,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:51,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:51,201 DEBUG [Thread-3278] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:51,201 DEBUG [Thread-3278] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:51,301 DEBUG [Thread-3278] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:51,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:51,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:51,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:51,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:51,304 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:51,305 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:51,305 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:51,305 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:51,305 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:51,305 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:51,307 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:51,307 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:51,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:51,307 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,308 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:51,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:51,308 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:51,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:51,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:51,308 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:51,308 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:51,309 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:51,309 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:51,309 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:51,309 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:51,309 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:51,309 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:51,309 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:51,309 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. started... 2023-03-31 16:55:51,309 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. as already flushing 2023-03-31 16:55:51,309 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:51,309 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:51,309 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:51,309 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,309 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:51,309 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:51,311 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:51,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:51,311 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:51,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:51,312 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:51,312 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:51,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:51,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:51,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:51,314 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:51,314 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f9b3f0b[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:51,314 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:51,314 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:51,315 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:51,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:51,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:51,315 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,315 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:51,315 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:51,315 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:51,315 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:51,317 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:51,317 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,317 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:51,317 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:51,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:51,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:51,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:51,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:51,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:51,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:51,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:51,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:51,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:51,319 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:51,319 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:51,319 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:51,319 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:51,319 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:51,321 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:51,321 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:51,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:51,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:51,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:51,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:51,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:51,321 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:51,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:51,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:51,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:51,322 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:51,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:51,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:51,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:51,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,323 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:51,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:51,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,327 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,327 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:51,327 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:51,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:51,327 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:51,327 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:51,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:51,327 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:51,327 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:51,327 DEBUG [Thread-3278] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:51,327 DEBUG [Thread-3278] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:51,327 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:51,327 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:51,327 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,327 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:51,327 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:51,327 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:51,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:51,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:51,391 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:51,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35580 deadline: 1680281811387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,427 DEBUG [Thread-3278] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:51,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:51,430 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:51,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:51,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:51,431 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:51,431 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:51,431 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:51,431 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:51,431 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:51,431 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:51,433 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,433 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:51,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:51,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:51,433 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,433 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:51,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:51,434 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:51,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:51,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:51,434 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:51,435 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:51,435 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:51,435 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:51,435 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:51,435 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:51,435 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:51,435 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:51,435 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:51,435 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. started... 2023-03-31 16:55:51,435 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. as already flushing 2023-03-31 16:55:51,435 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:51,436 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:51,436 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:51,436 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,436 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:51,436 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:51,437 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:51,437 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:51,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:51,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:51,438 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:51,438 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:51,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:51,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:51,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:51,439 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:51,439 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17d40ace[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:51,439 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:51,439 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:51,441 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:51,441 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,441 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:51,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:51,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:51,442 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:51,442 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:51,443 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:51,444 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,444 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:51,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:51,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:51,444 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:51,444 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:51,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:51,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:51,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:51,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:51,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:51,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:51,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:51,448 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:51,448 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:51,448 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:51,448 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:51,448 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:51,450 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:51,450 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:51,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:51,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:51,450 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:51,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:51,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:51,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:51,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:51,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:51,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:51,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:51,451 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:51,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:51,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:51,455 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:51,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:51,461 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,461 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:51,461 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:51,461 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,461 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:51,461 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:51,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,462 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:51,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:51,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:51,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:51,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:51,462 DEBUG [Thread-3278] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:51,462 DEBUG [Thread-3278] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:51,462 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:51,462 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:51,462 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:51,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:51,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:51,464 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:51,541 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=687 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/B/a9ca0c73aaec4967b43bfb0fc7ca016e 2023-03-31 16:55:51,553 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=687 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/C/b2bf5ac1a7484e68ae4427104f0644a6 2023-03-31 16:55:51,557 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/A/be75b33618904778a2f9484b9332aade as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/be75b33618904778a2f9484b9332aade 2023-03-31 16:55:51,563 DEBUG [Thread-3278] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:51,563 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/be75b33618904778a2f9484b9332aade, entries=200, sequenceid=687, filesize=14.3 K 2023-03-31 16:55:51,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:51,564 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/B/a9ca0c73aaec4967b43bfb0fc7ca016e as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/a9ca0c73aaec4967b43bfb0fc7ca016e 2023-03-31 16:55:51,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:51,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:51,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:51,570 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:51,570 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:51,570 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:51,570 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:51,572 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:51,572 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:51,576 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/a9ca0c73aaec4967b43bfb0fc7ca016e, entries=150, sequenceid=687, filesize=11.9 K 2023-03-31 16:55:51,579 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:51,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:51,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:51,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:51,579 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:51,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:51,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:51,580 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:51,580 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:51,580 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:51,580 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:51,580 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:51,580 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:51,580 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:51,580 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:51,581 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. started... 2023-03-31 16:55:51,580 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:51,581 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. as already flushing 2023-03-31 16:55:51,581 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:51,581 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:51,581 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:51,581 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,581 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:51,581 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:51,582 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,583 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:51,583 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/C/b2bf5ac1a7484e68ae4427104f0644a6 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/b2bf5ac1a7484e68ae4427104f0644a6 2023-03-31 16:55:51,583 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,583 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:51,584 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@564c1dab[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:51,584 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:51,584 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:51,584 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:51,584 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:51,584 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:51,585 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:51,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:51,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:51,585 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:51,585 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:51,585 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:51,585 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,586 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:51,586 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:51,586 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:51,586 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:51,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:51,587 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:51,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:51,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:51,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:51,588 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/b2bf5ac1a7484e68ae4427104f0644a6, entries=150, sequenceid=687, filesize=11.9 K 2023-03-31 16:55:51,588 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=46.96 KB/48090 for fd96548248d41c209e12fca5d7040af5 in 889ms, sequenceid=687, compaction requested=false 2023-03-31 16:55:51,588 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for fd96548248d41c209e12fca5d7040af5: 2023-03-31 16:55:51,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:51,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:51,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:51,590 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:51,590 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:51,590 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:51,590 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:51,590 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:51,591 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:51,598 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:51,598 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:51,598 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:51,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:51,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:51,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:51,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:51,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:51,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:51,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:51,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:51,598 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:51,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:51,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:51,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:51,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,599 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:51,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:51,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,604 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:51,604 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:51,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:51,604 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:51,604 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:51,604 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:51,605 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:51,605 DEBUG [Thread-3278] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:51,605 DEBUG [Thread-3278] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:51,604 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:51,605 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,604 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:51,605 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:51,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:51,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:51,605 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:51,605 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:51,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on fd96548248d41c209e12fca5d7040af5 2023-03-31 16:55:51,637 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing fd96548248d41c209e12fca5d7040af5 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-31 16:55:51,638 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=A 2023-03-31 16:55:51,638 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:51,638 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=B 2023-03-31 16:55:51,638 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:51,638 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=C 2023-03-31 16:55:51,638 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:51,684 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:51,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35494 deadline: 1680281811682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,685 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680281751675, max timestamp=1680281751683Num uniques 859; 2023-03-31 16:55:51,685 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680281751675, max timestamp=1680281751683Num uniques 859; 2023-03-31 16:55:51,685 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680281751675, max timestamp=1680281751683Num uniques 859; 2023-03-31 16:55:51,685 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 16:55:51,685 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 16:55:51,685 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 16:55:51,705 DEBUG [Thread-3278] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:51,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:51,707 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:51,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:51,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:51,708 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:51,708 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:51,708 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:51,708 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:51,708 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:51,708 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:51,709 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,709 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:51,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:51,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:51,709 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,709 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:51,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:51,710 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:51,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:51,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:51,710 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:51,711 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:51,711 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:51,711 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:51,711 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:51,711 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:51,711 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:51,711 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:51,711 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:51,711 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. started... 2023-03-31 16:55:51,711 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. as already flushing 2023-03-31 16:55:51,711 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:51,711 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:51,711 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:51,711 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,711 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:51,711 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:51,713 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:51,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:51,713 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:51,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:51,713 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:51,713 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:51,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:51,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:51,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:51,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:51,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52c8df91[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:51,714 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:51,714 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:51,716 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:51,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:51,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:51,716 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:51,717 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:51,717 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:51,717 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,717 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:51,718 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:51,718 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:51,718 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,718 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:51,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:51,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:51,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:51,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:51,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:51,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:51,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:51,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:51,720 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:51,720 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:51,720 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:51,720 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:51,720 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:51,720 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:51,722 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:51,722 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:51,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:51,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:51,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:51,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:51,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:51,722 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:51,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:51,722 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:51,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:51,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:51,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:51,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:51,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:51,723 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:51,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:51,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,724 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:51,724 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:51,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35582 deadline: 1680281811716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35496 deadline: 1680281811717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,725 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,725 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:51,725 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:51,725 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:51,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:51,726 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:51,726 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:51,726 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:51,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:51,726 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:51,726 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,726 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:51,726 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:51,726 DEBUG [Thread-3278] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:51,726 DEBUG [Thread-3278] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:51,726 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:51,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:51,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:51,730 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:51,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35546 deadline: 1680281811722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,791 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:51,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35494 deadline: 1680281811786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,827 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:51,827 DEBUG [Thread-3278] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:51,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35496 deadline: 1680281811826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:51,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:51,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:51,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:51,830 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:51,831 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:51,831 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:51,831 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:51,831 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:51,831 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:51,831 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:51,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35582 deadline: 1680281811826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,833 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:51,833 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:51,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:51,834 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,834 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:51,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:51,834 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:51,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:51,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:51,834 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:51,834 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:51,835 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:51,835 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:51,835 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:51,835 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:51,835 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:51,835 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:51,835 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:51,835 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. started... 2023-03-31 16:55:51,835 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. as already flushing 2023-03-31 16:55:51,835 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:51,835 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:51,835 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:51,835 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,835 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:51,835 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:51,836 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:51,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35546 deadline: 1680281811831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,837 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:51,837 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:51,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:51,837 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:51,837 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:51,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:51,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:51,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:51,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:51,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:51,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e11d4e0[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:51,838 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:51,838 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:51,840 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:51,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:51,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:51,840 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:51,840 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:51,840 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:51,840 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,840 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:51,843 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:51,843 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,843 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:51,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:51,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:51,843 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:51,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:51,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:51,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:51,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:51,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:51,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:51,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:51,845 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:51,845 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:51,845 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:51,845 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:51,845 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:51,847 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:51,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:51,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:51,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:51,847 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:51,848 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:51,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:51,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:51,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:51,848 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:51,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:51,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:51,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:51,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:51,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:51,849 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:51,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:51,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,851 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:51,851 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:51,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:51,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:51,851 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:51,851 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:51,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:51,851 DEBUG [Thread-3278] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:51,851 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:51,852 DEBUG [Thread-3278] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:51,852 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:51,852 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:51,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:51,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:51,852 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,852 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:51,852 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:51,852 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:51,899 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:51,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35580 deadline: 1680281811894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,952 DEBUG [Thread-3278] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:51,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:51,954 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:51,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:51,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:51,955 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:51,955 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:51,955 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:51,955 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:51,956 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:51,956 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:51,957 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:51,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:51,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:51,957 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:51,958 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:51,958 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,958 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:51,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:51,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:51,958 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:51,963 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:51,963 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:51,963 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:51,963 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:51,963 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:51,963 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:51,963 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:51,963 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:51,963 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. started... 2023-03-31 16:55:51,964 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. as already flushing 2023-03-31 16:55:51,964 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:51,964 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:51,964 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:51,964 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,964 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:51,964 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:51,966 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:51,966 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:51,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:51,966 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:51,966 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:51,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:51,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:51,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:51,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:51,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:51,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4864cb4e[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:51,970 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:51,970 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:51,972 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:51,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:51,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:51,972 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:51,973 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:51,973 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:51,973 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,973 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:51,974 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,974 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:51,974 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:51,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:51,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:51,974 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:51,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:51,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:51,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:51,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:51,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:51,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:51,978 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:51,978 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:51,978 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:51,978 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:51,978 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:51,978 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:51,980 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:51,980 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:51,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:51,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:51,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:51,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:51,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:51,980 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:51,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:51,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:51,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:51,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:51,982 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:51,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:51,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:51,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,983 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:51,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:51,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,992 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:51,992 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:51,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:51,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:51,992 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,992 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:51,992 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:51,992 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:51,992 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:51,992 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:51,992 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:51,992 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:51,992 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:51,993 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:51,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:51,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:51,993 DEBUG [Thread-3278] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:51,993 DEBUG [Thread-3278] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:51,995 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:51,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35494 deadline: 1680281811992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,032 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:52,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35496 deadline: 1680281812029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,039 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:52,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35582 deadline: 1680281812033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,042 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:52,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35546 deadline: 1680281812037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,046 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=701 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/A/408b46ee2c4f41ec9aea29a7e9bb7e90 2023-03-31 16:55:52,093 DEBUG [Thread-3278] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:52,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:52,095 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:52,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:52,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:52,096 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:52,096 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:52,096 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:52,096 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:52,096 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:52,096 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:52,098 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:52,098 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:52,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:52,098 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,098 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:52,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:52,099 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:52,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:52,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:52,099 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:52,099 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:52,099 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:52,099 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:52,099 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:52,099 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:52,099 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:52,100 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:52,100 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:52,100 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. started... 2023-03-31 16:55:52,100 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. as already flushing 2023-03-31 16:55:52,100 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:52,100 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:52,100 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:52,100 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,100 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:52,100 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:52,102 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,102 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:52,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:52,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:52,102 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:52,102 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:52,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:52,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:52,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:52,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:52,103 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:52,103 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7fd12af6[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:52,103 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:52,103 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:52,104 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:52,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:52,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:52,104 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:52,104 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:52,105 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:52,105 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,105 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:52,107 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,107 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:52,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:52,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:52,107 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:52,107 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:52,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:52,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:52,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:52,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:52,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:52,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:52,110 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:52,110 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:52,110 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:52,110 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:52,110 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:52,110 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:52,112 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:52,112 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:52,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:52,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:52,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:52,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:52,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:52,112 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:52,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:52,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:52,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:52,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:52,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:52,113 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:52,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:52,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:52,114 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:52,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,116 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,116 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:52,116 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:52,116 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,116 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:52,116 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:52,116 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:52,116 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:52,116 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:52,116 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:52,117 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:52,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:52,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:52,117 DEBUG [Thread-3278] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:52,117 DEBUG [Thread-3278] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:52,117 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:52,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:52,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:52,217 DEBUG [Thread-3278] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:52,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:52,219 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:52,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:52,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:52,220 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:52,220 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:52,220 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:52,220 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:52,220 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:52,220 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:52,222 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,222 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:52,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:52,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:52,222 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,222 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:52,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:52,222 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:52,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:52,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:52,223 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:52,223 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:52,223 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:52,223 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:52,223 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:52,223 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:52,223 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:52,223 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:52,223 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:52,223 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. started... 2023-03-31 16:55:52,223 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. as already flushing 2023-03-31 16:55:52,223 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:52,223 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:52,223 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:52,223 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,223 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:52,223 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:52,227 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:52,227 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:52,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:52,227 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:52,227 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:52,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:52,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:52,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:52,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:52,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:52,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b7156ca[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:52,228 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:52,228 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:52,229 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:52,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:52,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:52,230 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:52,230 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:52,230 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,230 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:52,230 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:52,231 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:52,231 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:52,231 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,231 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:52,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:52,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:52,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:52,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:52,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:52,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:52,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:52,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:52,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:52,233 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:52,234 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:52,234 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:52,234 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:52,234 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:52,235 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:52,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:52,235 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:52,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:52,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:52,235 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:52,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:52,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:52,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:52,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:52,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:52,236 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:52,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:52,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:52,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:52,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,237 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:52,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:52,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,240 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:52,240 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:52,240 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:52,240 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:52,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:52,240 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:52,241 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:52,240 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:52,241 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:52,241 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,241 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:52,241 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:52,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:52,241 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:52,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:52,241 DEBUG [Thread-3278] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:52,241 DEBUG [Thread-3278] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:52,297 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:52,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35494 deadline: 1680281812296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,340 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:52,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35496 deadline: 1680281812335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,341 DEBUG [Thread-3278] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:52,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:52,343 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:52,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:52,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:52,344 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:52,344 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:52,344 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:52,344 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:52,344 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:52,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35582 deadline: 1680281812340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,344 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:52,345 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:52,346 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:52,346 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:52,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:52,346 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:52,346 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:52,346 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:52,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:52,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:52,347 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:52,348 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:52,348 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:52,348 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:52,348 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:52,348 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:52,348 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:52,348 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:52,348 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:52,348 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. started... 2023-03-31 16:55:52,348 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. as already flushing 2023-03-31 16:55:52,348 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:52,348 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:52,348 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:52,348 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,348 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:52,348 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:52,350 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,350 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:52,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:52,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:52,350 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:52,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:52,350 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:52,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:52,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:52,351 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:52,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35546 deadline: 1680281812344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:52,351 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:52,351 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b973990[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:52,351 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:52,351 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:52,352 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:52,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:52,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:52,353 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:52,353 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,353 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:52,353 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:52,353 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:52,359 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,359 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:52,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:52,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:52,359 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:52,359 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:52,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:52,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:52,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:52,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:52,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:52,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:52,361 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:52,361 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:52,361 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:52,361 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:52,361 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:52,361 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:52,363 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:52,363 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:52,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:52,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:52,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:52,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:52,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:52,363 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:52,364 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:52,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:52,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:52,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:52,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:52,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:52,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:52,365 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:52,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:52,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,369 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,369 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:52,369 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:52,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:52,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:52,369 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:52,369 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:52,369 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:52,369 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,369 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:52,369 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:52,369 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:52,370 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:52,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:52,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:52,370 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:52,370 DEBUG [Thread-3278] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:52,370 DEBUG [Thread-3278] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:52,372 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:52,455 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=701 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/B/8f878fabbb954e4fa775f51764f84cf7 2023-03-31 16:55:52,465 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=701 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/C/7f704258fa4b49f893af6fb804382787 2023-03-31 16:55:52,469 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/A/408b46ee2c4f41ec9aea29a7e9bb7e90 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/408b46ee2c4f41ec9aea29a7e9bb7e90 2023-03-31 16:55:52,470 DEBUG [Thread-3278] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:52,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:52,472 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:52,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:52,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:52,473 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:52,473 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:52,473 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:52,473 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:52,474 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:52,474 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:52,475 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/408b46ee2c4f41ec9aea29a7e9bb7e90, entries=200, sequenceid=701, filesize=14.3 K 2023-03-31 16:55:52,475 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/B/8f878fabbb954e4fa775f51764f84cf7 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/8f878fabbb954e4fa775f51764f84cf7 2023-03-31 16:55:52,476 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:52,476 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:52,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:52,477 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,477 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:52,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:52,477 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:52,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:52,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:52,477 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:52,479 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/8f878fabbb954e4fa775f51764f84cf7, entries=150, sequenceid=701, filesize=11.9 K 2023-03-31 16:55:52,479 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:52,479 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/C/7f704258fa4b49f893af6fb804382787 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/7f704258fa4b49f893af6fb804382787 2023-03-31 16:55:52,480 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:52,480 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:52,480 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:52,480 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:52,480 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:52,480 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:52,480 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:52,480 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. started... 2023-03-31 16:55:52,481 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. as already flushing 2023-03-31 16:55:52,481 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:52,481 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:52,481 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:52,481 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,481 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:52,481 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:52,482 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,483 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:52,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:52,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:52,483 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:52,483 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:52,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:52,483 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/7f704258fa4b49f893af6fb804382787, entries=150, sequenceid=701, filesize=11.9 K 2023-03-31 16:55:52,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:52,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:52,484 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for fd96548248d41c209e12fca5d7040af5 in 847ms, sequenceid=701, compaction requested=true 2023-03-31 16:55:52,484 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for fd96548248d41c209e12fca5d7040af5: 2023-03-31 16:55:52,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,484 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:55:52,484 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 16:55:52,484 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 16:55:52,484 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:55:52,484 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:55:52,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:52,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:52,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5547bb17[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:52,485 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:52,485 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:52,485 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 41916 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 16:55:52,485 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1912): fd96548248d41c209e12fca5d7040af5/A is initiating minor compaction (all files) 2023-03-31 16:55:52,485 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of fd96548248d41c209e12fca5d7040af5/A in TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:52,485 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/a311336456b9497a86565ffef7c50086, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/be75b33618904778a2f9484b9332aade, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/408b46ee2c4f41ec9aea29a7e9bb7e90] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp, totalSize=40.9 K 2023-03-31 16:55:52,486 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37036 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 16:55:52,486 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): fd96548248d41c209e12fca5d7040af5/B is initiating minor compaction (all files) 2023-03-31 16:55:52,486 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fd96548248d41c209e12fca5d7040af5/B in TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:52,486 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/a323f04d9f0d484cbe71c78cbecdb1e4, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/a9ca0c73aaec4967b43bfb0fc7ca016e, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/8f878fabbb954e4fa775f51764f84cf7] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp, totalSize=36.2 K 2023-03-31 16:55:52,486 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:52,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:52,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:52,486 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting a311336456b9497a86565ffef7c50086, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=652, earliestPutTs=1680281749433 2023-03-31 16:55:52,486 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:52,486 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:52,486 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:52,487 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,487 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:52,487 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting a323f04d9f0d484cbe71c78cbecdb1e4, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=652, earliestPutTs=1680281749433 2023-03-31 16:55:52,487 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting be75b33618904778a2f9484b9332aade, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=687, earliestPutTs=1680281749519 2023-03-31 16:55:52,488 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting a9ca0c73aaec4967b43bfb0fc7ca016e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=687, earliestPutTs=1680281749519 2023-03-31 16:55:52,488 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,488 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:52,488 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:52,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:52,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:52,488 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:52,488 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 408b46ee2c4f41ec9aea29a7e9bb7e90, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=701, earliestPutTs=1680281750739 2023-03-31 16:55:52,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:52,489 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 8f878fabbb954e4fa775f51764f84cf7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=701, earliestPutTs=1680281750739 2023-03-31 16:55:52,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:52,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:52,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:52,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:52,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:52,491 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:52,491 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:52,491 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:52,491 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:52,491 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:52,491 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:52,494 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:52,494 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:52,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:52,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:52,494 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:52,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:52,495 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:52,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:52,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:52,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:52,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:52,495 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:52,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:52,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:52,497 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] throttle.PressureAwareThroughputController(145): fd96548248d41c209e12fca5d7040af5#A#compaction#479 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:55:52,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:52,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:52,498 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,498 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:52,498 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:52,498 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:52,498 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:52,498 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:52,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:52,499 DEBUG [Thread-3278] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:52,499 DEBUG [Thread-3278] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:52,498 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:52,499 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:52,498 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:52,499 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:52,499 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:52,499 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:52,499 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:52,499 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:52,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:52,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:52,501 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): fd96548248d41c209e12fca5d7040af5#B#compaction#480 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:55:52,520 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/A/8a49671736d149209dbb5b3b9dba8ada as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/8a49671736d149209dbb5b3b9dba8ada 2023-03-31 16:55:52,522 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/B/ec9c1ce3409245e6b22c33edac13ea86 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/ec9c1ce3409245e6b22c33edac13ea86 2023-03-31 16:55:52,530 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fd96548248d41c209e12fca5d7040af5/A of fd96548248d41c209e12fca5d7040af5 into 8a49671736d149209dbb5b3b9dba8ada(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:55:52,530 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for fd96548248d41c209e12fca5d7040af5: 2023-03-31 16:55:52,530 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5., storeName=fd96548248d41c209e12fca5d7040af5/A, priority=13, startTime=1680281752484; duration=0sec 2023-03-31 16:55:52,530 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:55:52,530 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 16:55:52,531 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fd96548248d41c209e12fca5d7040af5/B of fd96548248d41c209e12fca5d7040af5 into ec9c1ce3409245e6b22c33edac13ea86(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:55:52,532 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fd96548248d41c209e12fca5d7040af5: 2023-03-31 16:55:52,532 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5., storeName=fd96548248d41c209e12fca5d7040af5/B, priority=13, startTime=1680281752484; duration=0sec 2023-03-31 16:55:52,532 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:55:52,532 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37036 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 16:55:52,532 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1912): fd96548248d41c209e12fca5d7040af5/C is initiating minor compaction (all files) 2023-03-31 16:55:52,532 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of fd96548248d41c209e12fca5d7040af5/C in TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:52,532 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/0d1d9f722a6542ee9feaa02e210b8abc, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/b2bf5ac1a7484e68ae4427104f0644a6, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/7f704258fa4b49f893af6fb804382787] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp, totalSize=36.2 K 2023-03-31 16:55:52,532 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 0d1d9f722a6542ee9feaa02e210b8abc, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=652, earliestPutTs=1680281749433 2023-03-31 16:55:52,533 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting b2bf5ac1a7484e68ae4427104f0644a6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=687, earliestPutTs=1680281749519 2023-03-31 16:55:52,533 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 7f704258fa4b49f893af6fb804382787, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=701, earliestPutTs=1680281750739 2023-03-31 16:55:52,537 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] throttle.PressureAwareThroughputController(145): fd96548248d41c209e12fca5d7040af5#C#compaction#481 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:55:52,599 DEBUG [Thread-3278] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:52,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:52,601 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:52,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:52,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:52,602 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:52,602 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:52,602 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:52,602 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:52,603 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:52,603 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:52,604 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,604 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:52,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:52,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:52,604 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,604 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:52,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:52,605 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:52,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:52,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:52,605 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:52,605 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:52,605 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:52,606 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:52,606 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:52,606 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:52,606 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:52,606 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:52,606 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:52,606 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. started... 2023-03-31 16:55:52,606 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fd96548248d41c209e12fca5d7040af5 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-03-31 16:55:52,606 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=A 2023-03-31 16:55:52,606 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:55:52,606 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=B 2023-03-31 16:55:52,606 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:55:52,606 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=C 2023-03-31 16:55:52,606 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:55:52,614 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=738 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/A/36742c66d019482b8c8307f42de79c3c 2023-03-31 16:55:52,623 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=738 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/B/b7d99f8d1066467b8dea50a57d8b87c7 2023-03-31 16:55:52,637 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=738 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/C/fe0e4e08756b4b7092da9936163f236a 2023-03-31 16:55:52,643 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/A/36742c66d019482b8c8307f42de79c3c as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/36742c66d019482b8c8307f42de79c3c 2023-03-31 16:55:52,648 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/36742c66d019482b8c8307f42de79c3c, entries=150, sequenceid=738, filesize=11.9 K 2023-03-31 16:55:52,649 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/B/b7d99f8d1066467b8dea50a57d8b87c7 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/b7d99f8d1066467b8dea50a57d8b87c7 2023-03-31 16:55:52,653 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/b7d99f8d1066467b8dea50a57d8b87c7, entries=150, sequenceid=738, filesize=11.9 K 2023-03-31 16:55:52,659 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/C/fe0e4e08756b4b7092da9936163f236a as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/fe0e4e08756b4b7092da9936163f236a 2023-03-31 16:55:52,663 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/fe0e4e08756b4b7092da9936163f236a, entries=150, sequenceid=738, filesize=11.9 K 2023-03-31 16:55:52,664 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=0 B/0 for fd96548248d41c209e12fca5d7040af5 in 58ms, sequenceid=738, compaction requested=false 2023-03-31 16:55:52,664 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fd96548248d41c209e12fca5d7040af5: 2023-03-31 16:55:52,664 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:52,664 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:52,664 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:52,664 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,665 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:52,665 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:52,666 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,666 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:52,667 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:52,667 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:52,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:52,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:52,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:52,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:52,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:52,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:52,669 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:52,669 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23b7bdd3[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:52,669 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:52,669 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:52,675 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:52,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:52,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:52,676 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:52,676 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:52,676 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:52,677 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,677 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:52,677 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,677 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:52,677 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:52,678 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:52,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:52,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:52,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:52,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:52,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:52,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:52,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:52,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:52,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:52,680 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:52,680 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:52,680 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:52,680 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:52,680 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:52,683 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:52,683 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:52,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:52,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:52,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:52,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:52,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:52,683 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:52,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:52,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:52,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:52,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:52,684 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:52,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:52,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:52,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:52,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:52,686 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,696 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:52,696 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,696 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:52,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:52,696 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:52,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:52,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:52,696 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:52,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:52,696 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:52,696 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,696 DEBUG [Thread-3278] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:52,697 DEBUG [Thread-3278] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:52,696 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:52,697 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:52,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:52,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:52,697 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:52,697 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:52,797 DEBUG [Thread-3278] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:52,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:52,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:52,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:52,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:52,804 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:52,804 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:52,804 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:52,805 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:52,806 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:52,806 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:52,808 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:52,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:52,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:52,808 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:52,809 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:52,809 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,809 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:52,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:52,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:52,810 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:52,810 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:52,810 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:52,810 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:52,810 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:52,811 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:52,811 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:52,811 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:52,811 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:52,811 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. started... 2023-03-31 16:55:52,811 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fd96548248d41c209e12fca5d7040af5 3/3 column families, dataSize=13.42 KB heapSize=40.59 KB 2023-03-31 16:55:52,811 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=A 2023-03-31 16:55:52,811 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:52,811 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=B 2023-03-31 16:55:52,812 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:52,812 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=C 2023-03-31 16:55:52,812 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:52,826 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=743 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/A/196c3e427396455aaa23cd195ebd0802 2023-03-31 16:55:52,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on fd96548248d41c209e12fca5d7040af5 2023-03-31 16:55:52,832 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. as already flushing 2023-03-31 16:55:52,852 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=743 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/B/228bc33de14f481390094dee19dbf242 2023-03-31 16:55:52,864 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=743 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/C/64a381552e44443dacb98d2591a749a5 2023-03-31 16:55:52,867 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14144, totalHeapSize=32376, min timestamp=1680281752857, max timestamp=1680281752865Num uniques 859; 2023-03-31 16:55:52,867 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=14144, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-31 16:55:52,867 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14144, totalHeapSize=32376, min timestamp=1680281752857, max timestamp=1680281752865Num uniques 859; 2023-03-31 16:55:52,867 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14144, totalHeapSize=32376, min timestamp=1680281752857, max timestamp=1680281752865Num uniques 859; 2023-03-31 16:55:52,868 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=14144, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-31 16:55:52,867 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=14144, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-31 16:55:52,869 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/A/196c3e427396455aaa23cd195ebd0802 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/196c3e427396455aaa23cd195ebd0802 2023-03-31 16:55:52,872 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/196c3e427396455aaa23cd195ebd0802, entries=100, sequenceid=743, filesize=9.5 K 2023-03-31 16:55:52,873 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/B/228bc33de14f481390094dee19dbf242 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/228bc33de14f481390094dee19dbf242 2023-03-31 16:55:52,876 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/228bc33de14f481390094dee19dbf242, entries=100, sequenceid=743, filesize=9.5 K 2023-03-31 16:55:52,876 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/C/64a381552e44443dacb98d2591a749a5 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/64a381552e44443dacb98d2591a749a5 2023-03-31 16:55:52,880 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/64a381552e44443dacb98d2591a749a5, entries=100, sequenceid=743, filesize=9.5 K 2023-03-31 16:55:52,880 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~13.42 KB/13740, heapSize ~40.55 KB/41520, currentSize=80.51 KB/82440 for fd96548248d41c209e12fca5d7040af5 in 69ms, sequenceid=743, compaction requested=true 2023-03-31 16:55:52,880 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fd96548248d41c209e12fca5d7040af5: 2023-03-31 16:55:52,880 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:52,880 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:52,880 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:52,880 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,880 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:52,881 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:52,882 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:52,882 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:52,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:52,882 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:52,882 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:52,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:52,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:52,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:52,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on fd96548248d41c209e12fca5d7040af5 2023-03-31 16:55:52,883 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing fd96548248d41c209e12fca5d7040af5 3/3 column families, dataSize=87.22 KB heapSize=231.47 KB 2023-03-31 16:55:52,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,883 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=A 2023-03-31 16:55:52,884 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:55:52,884 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=B 2023-03-31 16:55:52,884 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:55:52,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:52,884 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=C 2023-03-31 16:55:52,884 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:55:52,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:52,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42916869[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:52,884 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:52,884 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:52,885 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:52,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:52,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:52,887 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,887 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:52,887 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:52,887 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:52,887 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:52,889 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:52,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:52,889 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:52,889 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:52,889 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:52,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:52,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:52,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:52,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:52,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:52,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:52,892 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:52,892 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:52,892 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:52,892 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:52,892 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:52,892 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:52,894 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:52,894 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:52,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:52,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:52,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:52,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:52,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:52,894 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:52,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:52,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:52,895 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:52,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:52,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:52,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:52,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:52,896 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:52,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:52,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,898 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:52,898 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:52,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:52,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:52,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:52,899 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,899 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:52,899 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:52,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:52,899 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:52,899 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,899 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:52,899 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:52,899 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:52,899 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:52,899 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:52,899 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:52,899 DEBUG [Thread-3278] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:52,899 DEBUG [Thread-3278] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:52,933 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:52,933 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:52,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35496 deadline: 1680281812918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35494 deadline: 1680281812918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,934 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:52,934 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:52,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35546 deadline: 1680281812919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35580 deadline: 1680281812919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,940 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:52,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35582 deadline: 1680281812932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:52,944 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/C/5d31905e36334081acef1152f9f95857 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/5d31905e36334081acef1152f9f95857 2023-03-31 16:55:52,947 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fd96548248d41c209e12fca5d7040af5/C of fd96548248d41c209e12fca5d7040af5 into 5d31905e36334081acef1152f9f95857(size=12.5 K), total size for store is 33.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:55:52,948 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for fd96548248d41c209e12fca5d7040af5: 2023-03-31 16:55:52,948 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5., storeName=fd96548248d41c209e12fca5d7040af5/C, priority=13, startTime=1680281752484; duration=0sec 2023-03-31 16:55:52,948 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:55:52,999 DEBUG [Thread-3278] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:53,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:53,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:53,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:53,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:53,003 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:53,003 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:53,003 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:53,003 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:53,004 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:53,004 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:53,005 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:53,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:53,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:53,005 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:53,006 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:53,006 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,006 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:53,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:53,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:53,006 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:53,007 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:53,007 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:53,007 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:53,007 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:53,007 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:53,007 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:53,007 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:53,008 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:53,008 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. started... 2023-03-31 16:55:53,008 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. as already flushing 2023-03-31 16:55:53,008 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:53,008 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:53,008 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:53,008 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,008 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:53,008 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:53,014 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,014 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:53,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:53,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:53,014 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:53,014 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:53,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:53,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:53,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:53,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:53,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:53,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40488212[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:53,016 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:53,016 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:53,018 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:53,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:53,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:53,019 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:53,019 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:53,019 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:53,019 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,019 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:53,020 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,020 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:53,020 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:53,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:53,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:53,021 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:53,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:53,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:53,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:53,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:53,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:53,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:53,029 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:53,029 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:53,029 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:53,029 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:53,029 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:53,029 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:53,031 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:53,031 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:53,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:53,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:53,031 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:53,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:53,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:53,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:53,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:53,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:53,031 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:53,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:53,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:53,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:53,033 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:53,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:53,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:53,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,038 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,038 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:53,038 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:53,038 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,038 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:53,038 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:53,038 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:53,038 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:53,038 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:53,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:53,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:53,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:53,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:53,038 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:53,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:53,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:53,042 DEBUG [Thread-3278] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:53,042 DEBUG [Thread-3278] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:53,046 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:53,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35494 deadline: 1680281813034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,046 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:53,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35496 deadline: 1680281813035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,046 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:53,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35546 deadline: 1680281813035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,046 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:53,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35580 deadline: 1680281813035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,047 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:53,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35582 deadline: 1680281813041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,142 DEBUG [Thread-3278] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:53,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:53,144 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:53,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:53,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:53,145 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:53,145 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:53,145 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:53,145 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:53,145 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:53,145 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:53,147 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:53,147 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:53,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:53,147 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,147 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:53,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:53,148 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:53,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:53,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:53,148 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:53,148 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:53,148 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:53,148 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:53,148 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:53,148 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:53,148 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:53,149 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:53,149 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:53,149 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. started... 2023-03-31 16:55:53,149 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. as already flushing 2023-03-31 16:55:53,149 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:53,149 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:53,149 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:53,149 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,149 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:53,149 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:53,150 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,150 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:53,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:53,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:53,151 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:53,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:53,151 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:53,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:53,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:53,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:53,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:53,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@283f060f[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:53,152 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:53,152 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:53,153 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:53,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:53,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:53,153 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:53,153 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:53,153 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:53,153 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,153 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:53,155 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:53,156 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:53,156 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:53,156 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:53,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:53,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:53,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:53,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:53,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:53,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:53,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:53,158 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:53,158 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:53,158 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:53,158 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:53,158 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:53,158 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:53,160 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:53,160 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:53,160 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:53,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:53,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:53,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:53,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:53,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:53,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:53,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:53,161 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:53,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:53,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:53,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:53,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:53,164 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:53,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:53,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,166 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,166 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:53,166 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:53,166 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:53,166 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:53,166 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:53,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:53,166 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:53,167 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:53,167 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:53,166 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:53,166 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:53,167 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:53,167 DEBUG [Thread-3278] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:53,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:53,167 DEBUG [Thread-3278] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:53,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:53,254 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:53,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35580 deadline: 1680281813247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,254 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:53,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35546 deadline: 1680281813248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,255 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:53,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35496 deadline: 1680281813249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,258 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:53,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35494 deadline: 1680281813253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,258 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:53,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35582 deadline: 1680281813253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,267 DEBUG [Thread-3278] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:53,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:53,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:53,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:53,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:53,270 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:53,270 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:53,270 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:53,270 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:53,271 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:53,271 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:53,272 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:53,272 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:53,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:53,272 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:53,272 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:53,273 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:53,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:53,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:53,273 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:53,273 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:53,273 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:53,273 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:53,273 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:53,273 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:53,273 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:53,273 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:53,274 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:53,274 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. started... 2023-03-31 16:55:53,274 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. as already flushing 2023-03-31 16:55:53,274 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:53,274 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:53,274 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:53,274 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,274 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:53,274 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:53,277 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,277 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:53,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:53,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:53,277 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:53,277 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:53,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:53,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:53,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:53,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:53,278 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:53,278 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36787ffe[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:53,278 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:53,278 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:53,279 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:53,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:53,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:53,280 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:53,280 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:53,280 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:53,280 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,280 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:53,281 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:53,281 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,281 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:53,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:53,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:53,281 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:53,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:53,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:53,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:53,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:53,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:53,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:53,283 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:53,283 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:53,283 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:53,283 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:53,283 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:53,283 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:53,285 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:53,285 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:53,285 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:53,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:53,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:53,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:53,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:53,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:53,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:53,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:53,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:53,286 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:53,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:53,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:53,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:53,287 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:53,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:53,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,290 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,290 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:53,290 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:53,290 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,290 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:53,290 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:53,290 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:53,290 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:53,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:53,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:53,290 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:53,290 DEBUG [Thread-3278] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:53,290 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:53,290 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:53,290 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:53,290 DEBUG [Thread-3278] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:53,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:53,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:53,291 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=770 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/A/2617ae3f4d7a44d585c8512e9eaae28e 2023-03-31 16:55:53,391 DEBUG [Thread-3278] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:53,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:53,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:53,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:53,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:53,394 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:53,394 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:53,394 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:53,394 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:53,394 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:53,394 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:53,395 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,395 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:53,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:53,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:53,396 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,396 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:53,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:53,396 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:53,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:53,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:53,397 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:53,397 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:53,397 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:53,397 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:53,397 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:53,397 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:53,397 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:53,397 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:53,397 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:53,397 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. started... 2023-03-31 16:55:53,397 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. as already flushing 2023-03-31 16:55:53,397 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:53,397 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:53,398 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:53,398 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,398 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:53,398 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:53,399 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,399 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:53,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:53,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:53,399 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:53,399 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:53,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:53,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:53,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:53,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:53,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:53,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f638b78[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:53,401 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:53,401 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:53,406 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:53,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:53,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:53,406 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:53,406 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:53,406 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:53,406 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,406 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:53,407 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,407 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:53,407 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:53,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:53,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:53,407 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:53,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:53,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:53,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:53,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:53,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:53,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:53,409 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:53,409 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:53,410 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:53,410 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:53,410 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:53,410 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:53,411 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:53,411 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:53,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:53,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:53,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:53,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:53,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:53,411 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:53,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:53,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:53,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:53,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:53,412 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:53,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:53,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:53,413 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:53,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:53,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,415 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,415 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:53,415 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:53,415 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,415 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:53,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:53,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:53,415 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:53,415 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:53,415 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:53,415 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:53,415 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:53,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:53,415 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:53,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:53,415 DEBUG [Thread-3278] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:53,416 DEBUG [Thread-3278] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:53,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:53,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:53,516 DEBUG [Thread-3278] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:53,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:53,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:53,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:53,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:53,519 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:53,519 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:53,519 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:53,519 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:53,519 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:53,520 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:53,522 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,522 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:53,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:53,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:53,522 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,522 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:53,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:53,522 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:53,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:53,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:53,523 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:53,524 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:53,524 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:53,524 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:53,524 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:53,524 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:53,524 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:53,524 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:53,524 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. started... 2023-03-31 16:55:53,524 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. as already flushing 2023-03-31 16:55:53,524 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:53,524 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:53,525 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:53,525 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,525 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:53,525 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:53,526 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,526 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:53,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:53,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:53,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:53,527 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:53,527 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:53,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:53,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:53,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:53,528 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:53,528 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7bbfde94[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:53,528 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:53,528 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:53,529 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:53,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:53,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:53,530 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:53,530 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:53,530 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:53,530 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,530 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:53,531 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,531 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:53,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:53,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:53,531 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:53,531 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:53,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:53,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:53,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:53,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:53,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:53,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:53,537 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:53,537 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:53,537 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:53,537 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:53,537 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:53,537 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:53,540 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:53,540 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:53,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:53,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:53,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:53,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:53,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:53,540 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:53,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:53,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:53,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:53,540 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:53,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:53,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:53,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:53,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,542 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:53,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:53,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,545 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,545 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:53,545 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:53,545 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:53,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:53,545 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:53,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:53,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:53,545 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:53,545 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,545 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:53,545 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:53,545 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:53,545 DEBUG [Thread-3278] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:53,545 DEBUG [Thread-3278] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:53,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:53,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:53,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:53,566 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:53,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35580 deadline: 1680281813557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,567 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:53,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35546 deadline: 1680281813557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,567 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:53,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35496 deadline: 1680281813558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,567 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:53,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35494 deadline: 1680281813561, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,568 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:53,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35582 deadline: 1680281813562, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,645 DEBUG [Thread-3278] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:53,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:53,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:53,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:53,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:53,656 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:53,656 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:53,656 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:53,656 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:53,656 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:53,657 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:53,658 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,658 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:53,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:53,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:53,658 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:53,658 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:53,659 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:53,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:53,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:53,659 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:53,676 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:53,676 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:53,676 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:53,676 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:53,676 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:53,688 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:53,688 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:53,691 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:53,691 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. started... 2023-03-31 16:55:53,691 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. as already flushing 2023-03-31 16:55:53,692 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:53,692 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:53,692 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:53,692 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,692 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:53,692 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:53,694 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,694 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:53,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:53,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:53,694 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:53,694 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:53,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:53,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:53,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:53,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:53,696 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:53,696 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b60a092[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:53,696 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:53,696 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:53,697 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:53,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:53,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:53,697 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:53,697 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:53,697 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:53,698 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,698 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:53,700 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:53,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:53,700 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:53,700 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:53,700 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:53,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:53,701 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=770 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/B/0cf3ffffde0541c6bb2eb23a85eae8ab 2023-03-31 16:55:53,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:53,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:53,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:53,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:53,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:53,705 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:53,705 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:53,705 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:53,705 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:53,705 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:53,705 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:53,707 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:53,707 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:53,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:53,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:53,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:53,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:53,707 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:53,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:53,707 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:53,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:53,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:53,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:53,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:53,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:53,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:53,708 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:53,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:53,711 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,711 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:53,711 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:53,711 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,711 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:53,711 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:53,711 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:53,711 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:53,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:53,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:53,711 DEBUG [Thread-3278] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:53,711 DEBUG [Thread-3278] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:53,711 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:53,711 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:53,711 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:53,711 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:53,711 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:53,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:53,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:53,744 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=770 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/C/c290e38f115345dc989b8300f91780b6 2023-03-31 16:55:53,749 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/A/2617ae3f4d7a44d585c8512e9eaae28e as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/2617ae3f4d7a44d585c8512e9eaae28e 2023-03-31 16:55:53,753 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/2617ae3f4d7a44d585c8512e9eaae28e, entries=300, sequenceid=770, filesize=19.0 K 2023-03-31 16:55:53,754 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/B/0cf3ffffde0541c6bb2eb23a85eae8ab as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/0cf3ffffde0541c6bb2eb23a85eae8ab 2023-03-31 16:55:53,757 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/0cf3ffffde0541c6bb2eb23a85eae8ab, entries=150, sequenceid=770, filesize=11.9 K 2023-03-31 16:55:53,758 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/C/c290e38f115345dc989b8300f91780b6 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/c290e38f115345dc989b8300f91780b6 2023-03-31 16:55:53,765 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/c290e38f115345dc989b8300f91780b6, entries=150, sequenceid=770, filesize=11.9 K 2023-03-31 16:55:53,765 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~231.42 KB/236976, currentSize=100.63 KB/103050 for fd96548248d41c209e12fca5d7040af5 in 882ms, sequenceid=770, compaction requested=true 2023-03-31 16:55:53,765 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for fd96548248d41c209e12fca5d7040af5: 2023-03-31 16:55:53,766 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:55:53,766 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:55:53,766 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 16:55:53,766 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:55:53,766 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 16:55:53,776 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 54200 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 16:55:53,776 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1912): fd96548248d41c209e12fca5d7040af5/A is initiating minor compaction (all files) 2023-03-31 16:55:53,776 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of fd96548248d41c209e12fca5d7040af5/A in TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:53,776 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/8a49671736d149209dbb5b3b9dba8ada, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/36742c66d019482b8c8307f42de79c3c, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/196c3e427396455aaa23cd195ebd0802, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/2617ae3f4d7a44d585c8512e9eaae28e] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp, totalSize=52.9 K 2023-03-31 16:55:53,776 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46880 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 16:55:53,776 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): fd96548248d41c209e12fca5d7040af5/B is initiating minor compaction (all files) 2023-03-31 16:55:53,776 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fd96548248d41c209e12fca5d7040af5/B in TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:53,777 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/ec9c1ce3409245e6b22c33edac13ea86, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/b7d99f8d1066467b8dea50a57d8b87c7, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/228bc33de14f481390094dee19dbf242, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/0cf3ffffde0541c6bb2eb23a85eae8ab] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp, totalSize=45.8 K 2023-03-31 16:55:53,777 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 8a49671736d149209dbb5b3b9dba8ada, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=701, earliestPutTs=1680281750739 2023-03-31 16:55:53,777 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting ec9c1ce3409245e6b22c33edac13ea86, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=701, earliestPutTs=1680281750739 2023-03-31 16:55:53,777 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 36742c66d019482b8c8307f42de79c3c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=738, earliestPutTs=1680281751715 2023-03-31 16:55:53,777 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting b7d99f8d1066467b8dea50a57d8b87c7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=738, earliestPutTs=1680281751715 2023-03-31 16:55:53,777 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 196c3e427396455aaa23cd195ebd0802, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=743, earliestPutTs=1680281752806 2023-03-31 16:55:53,778 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 228bc33de14f481390094dee19dbf242, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=743, earliestPutTs=1680281752806 2023-03-31 16:55:53,778 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 2617ae3f4d7a44d585c8512e9eaae28e, keycount=300, bloomtype=ROW, size=19.0 K, encoding=NONE, compression=NONE, seqNum=770, earliestPutTs=1680281752864 2023-03-31 16:55:53,778 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 0cf3ffffde0541c6bb2eb23a85eae8ab, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=770, earliestPutTs=1680281752876 2023-03-31 16:55:53,794 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] throttle.PressureAwareThroughputController(145): fd96548248d41c209e12fca5d7040af5#A#compaction#491 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:55:53,797 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): fd96548248d41c209e12fca5d7040af5#B#compaction#492 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:55:53,811 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/A/1ca8bc1c03fc4bb8ab992fe817a203c6 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/1ca8bc1c03fc4bb8ab992fe817a203c6 2023-03-31 16:55:53,811 DEBUG [Thread-3278] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:53,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:53,816 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:53,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:53,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:53,817 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in fd96548248d41c209e12fca5d7040af5/A of fd96548248d41c209e12fca5d7040af5 into 1ca8bc1c03fc4bb8ab992fe817a203c6(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:55:53,818 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for fd96548248d41c209e12fca5d7040af5: 2023-03-31 16:55:53,818 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5., storeName=fd96548248d41c209e12fca5d7040af5/A, priority=12, startTime=1680281753766; duration=0sec 2023-03-31 16:55:53,818 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:55:53,818 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 16:55:53,818 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:53,818 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:53,818 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:53,818 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:53,818 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:53,819 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:53,819 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46880 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 16:55:53,819 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1912): fd96548248d41c209e12fca5d7040af5/C is initiating minor compaction (all files) 2023-03-31 16:55:53,819 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of fd96548248d41c209e12fca5d7040af5/C in TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:53,819 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/5d31905e36334081acef1152f9f95857, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/fe0e4e08756b4b7092da9936163f236a, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/64a381552e44443dacb98d2591a749a5, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/c290e38f115345dc989b8300f91780b6] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp, totalSize=45.8 K 2023-03-31 16:55:53,820 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 5d31905e36334081acef1152f9f95857, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=701, earliestPutTs=1680281750739 2023-03-31 16:55:53,820 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:53,820 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:53,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:53,820 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting fe0e4e08756b4b7092da9936163f236a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=738, earliestPutTs=1680281751715 2023-03-31 16:55:53,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:53,820 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:53,821 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:53,821 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:53,821 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 64a381552e44443dacb98d2591a749a5, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=743, earliestPutTs=1680281752806 2023-03-31 16:55:53,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:53,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:53,821 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting c290e38f115345dc989b8300f91780b6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=770, earliestPutTs=1680281752876 2023-03-31 16:55:53,821 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:53,821 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:53,821 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:53,822 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:53,822 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:53,822 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:53,822 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:53,822 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:53,822 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:53,822 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. started... 2023-03-31 16:55:53,822 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fd96548248d41c209e12fca5d7040af5 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-31 16:55:53,822 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=A 2023-03-31 16:55:53,822 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:53,822 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=B 2023-03-31 16:55:53,822 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:53,822 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=C 2023-03-31 16:55:53,822 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:53,837 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] throttle.PressureAwareThroughputController(145): fd96548248d41c209e12fca5d7040af5#C#compaction#494 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:55:53,874 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/C/ac043966fc7448c0972e50fb65d82bf7 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/ac043966fc7448c0972e50fb65d82bf7 2023-03-31 16:55:53,879 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in fd96548248d41c209e12fca5d7040af5/C of fd96548248d41c209e12fca5d7040af5 into ac043966fc7448c0972e50fb65d82bf7(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:55:53,879 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for fd96548248d41c209e12fca5d7040af5: 2023-03-31 16:55:53,879 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5., storeName=fd96548248d41c209e12fca5d7040af5/C, priority=12, startTime=1680281753766; duration=0sec 2023-03-31 16:55:53,879 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:55:54,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on fd96548248d41c209e12fca5d7040af5 2023-03-31 16:55:54,072 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. as already flushing 2023-03-31 16:55:54,088 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:54,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35494 deadline: 1680281814083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:54,089 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:54,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35582 deadline: 1680281814086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:54,092 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:54,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35496 deadline: 1680281814087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:54,098 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:54,098 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:54,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35546 deadline: 1680281814089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:54,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35580 deadline: 1680281814089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:54,194 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:54,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35494 deadline: 1680281814190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:54,194 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:54,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35582 deadline: 1680281814190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:54,198 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:54,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35496 deadline: 1680281814193, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:54,202 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:54,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35580 deadline: 1680281814199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:54,205 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:54,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35546 deadline: 1680281814200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:54,213 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/B/cfb5ffd9bdbb42018e249743caa25b00 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/cfb5ffd9bdbb42018e249743caa25b00 2023-03-31 16:55:54,218 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in fd96548248d41c209e12fca5d7040af5/B of fd96548248d41c209e12fca5d7040af5 into cfb5ffd9bdbb42018e249743caa25b00(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:55:54,218 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fd96548248d41c209e12fca5d7040af5: 2023-03-31 16:55:54,218 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5., storeName=fd96548248d41c209e12fca5d7040af5/B, priority=12, startTime=1680281753766; duration=0sec 2023-03-31 16:55:54,219 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:55:54,269 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=790 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/A/834a0c767616459084dfac42dd3ad093 2023-03-31 16:55:54,280 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=790 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/B/03352426d4e543c2a58b4f37f24fa868 2023-03-31 16:55:54,398 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:54,398 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:54,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35494 deadline: 1680281814396, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:54,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35582 deadline: 1680281814396, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:54,404 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:54,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35496 deadline: 1680281814400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:54,408 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:54,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35580 deadline: 1680281814405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:54,409 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:54,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35546 deadline: 1680281814407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:54,707 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:54,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35582 deadline: 1680281814701, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:54,707 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:54,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35494 deadline: 1680281814702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:54,710 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=790 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/C/5de85fbfe91646239c8619e33f83e550 2023-03-31 16:55:54,712 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:54,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35496 deadline: 1680281814707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:54,712 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:54,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35546 deadline: 1680281814710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:54,712 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:54,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35580 deadline: 1680281814711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:54,715 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/A/834a0c767616459084dfac42dd3ad093 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/834a0c767616459084dfac42dd3ad093 2023-03-31 16:55:54,718 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/834a0c767616459084dfac42dd3ad093, entries=150, sequenceid=790, filesize=11.9 K 2023-03-31 16:55:54,719 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/B/03352426d4e543c2a58b4f37f24fa868 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/03352426d4e543c2a58b4f37f24fa868 2023-03-31 16:55:54,722 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/03352426d4e543c2a58b4f37f24fa868, entries=150, sequenceid=790, filesize=11.9 K 2023-03-31 16:55:54,723 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/C/5de85fbfe91646239c8619e33f83e550 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/5de85fbfe91646239c8619e33f83e550 2023-03-31 16:55:54,726 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/5de85fbfe91646239c8619e33f83e550, entries=150, sequenceid=790, filesize=11.9 K 2023-03-31 16:55:54,726 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=80.51 KB/82440 for fd96548248d41c209e12fca5d7040af5 in 904ms, sequenceid=790, compaction requested=false 2023-03-31 16:55:54,726 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fd96548248d41c209e12fca5d7040af5: 2023-03-31 16:55:54,726 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:54,727 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:54,727 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:54,727 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:54,728 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:54,728 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:54,732 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:54,732 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:54,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:54,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:54,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:54,732 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:54,732 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:54,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:54,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:54,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:54,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:54,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:54,746 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:54,746 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e12bd8f[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:54,746 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:54,746 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:54,747 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:54,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:54,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:54,748 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:54,748 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:54,748 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:54,748 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:54,748 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:54,751 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:54,751 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:54,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:54,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:54,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:54,752 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:54,752 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:54,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:54,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:54,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:54,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:54,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:54,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:54,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:54,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:54,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:54,754 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:54,754 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:54,754 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:54,754 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:54,754 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:54,757 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:54,757 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:54,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:54,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:54,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:54,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:54,757 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:54,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:54,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:54,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:54,758 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:54,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:54,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:54,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:54,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:54,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:54,759 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:54,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:54,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:54,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:54,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:54,762 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:54,762 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:54,762 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:54,762 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:54,762 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:54,762 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:54,762 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:54,762 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:54,762 DEBUG [Thread-3278] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:54,762 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:54,762 DEBUG [Thread-3278] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:54,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:54,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:54,762 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:54,763 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:54,763 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:54,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:54,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:54,863 DEBUG [Thread-3278] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:54,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:54,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:54,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:54,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:54,865 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:54,865 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:54,866 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:54,866 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:54,866 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:54,866 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:54,867 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:54,867 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:54,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:54,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:54,868 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:54,868 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:54,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:54,868 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:54,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:54,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:54,869 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:54,869 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:54,869 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:54,869 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:54,869 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:54,869 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:54,870 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:54,870 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:54,870 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:54,870 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. started... 2023-03-31 16:55:54,870 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fd96548248d41c209e12fca5d7040af5 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-31 16:55:54,870 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=A 2023-03-31 16:55:54,870 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:54,871 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=B 2023-03-31 16:55:54,871 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:54,871 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=C 2023-03-31 16:55:54,871 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:54,878 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=807 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/A/9b62fccf96fc4ea2bb4f07713b154338 2023-03-31 16:55:55,215 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. as already flushing 2023-03-31 16:55:55,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on fd96548248d41c209e12fca5d7040af5 2023-03-31 16:55:55,273 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:55,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35546 deadline: 1680281815237, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:55,282 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:55,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35496 deadline: 1680281815248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:55,283 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:55,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35582 deadline: 1680281815273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:55,283 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:55,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35494 deadline: 1680281815273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:55,284 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:55,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35580 deadline: 1680281815273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:55,289 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=807 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/B/f667690121dc48cf8269b6482f058333 2023-03-31 16:55:55,338 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=807 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/C/17044945c73e48ec8927e37589b03625 2023-03-31 16:55:55,342 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/A/9b62fccf96fc4ea2bb4f07713b154338 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/9b62fccf96fc4ea2bb4f07713b154338 2023-03-31 16:55:55,346 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/9b62fccf96fc4ea2bb4f07713b154338, entries=150, sequenceid=807, filesize=11.9 K 2023-03-31 16:55:55,347 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/B/f667690121dc48cf8269b6482f058333 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/f667690121dc48cf8269b6482f058333 2023-03-31 16:55:55,351 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/f667690121dc48cf8269b6482f058333, entries=150, sequenceid=807, filesize=11.9 K 2023-03-31 16:55:55,352 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/C/17044945c73e48ec8927e37589b03625 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/17044945c73e48ec8927e37589b03625 2023-03-31 16:55:55,361 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/17044945c73e48ec8927e37589b03625, entries=150, sequenceid=807, filesize=11.9 K 2023-03-31 16:55:55,362 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=107.34 KB/109920 for fd96548248d41c209e12fca5d7040af5 in 492ms, sequenceid=807, compaction requested=true 2023-03-31 16:55:55,362 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fd96548248d41c209e12fca5d7040af5: 2023-03-31 16:55:55,362 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:55,362 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:55,362 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:55,362 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:55,363 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:55,363 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:55,364 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:55,365 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:55,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:55,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:55,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:55,365 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:55,365 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:55,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:55,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:55,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:55,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:55,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:55,366 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:55,366 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3dc9c57e[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:55,366 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:55,367 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:55,368 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:55,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:55,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:55,368 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:55,368 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:55,368 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:55,369 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:55,369 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:55,371 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:55,371 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:55,371 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:55,371 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:55,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:55,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:55,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:55,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:55,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:55,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:55,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:55,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:55,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:55,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:55,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:55,374 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:55,374 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:55,374 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:55,374 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:55,374 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:55,374 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:55,376 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:55,376 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:55,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:55,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:55,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:55,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:55,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:55,377 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:55,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:55,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:55,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:55,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:55,378 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:55,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:55,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:55,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:55,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:55,379 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:55,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:55,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:55,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:55,382 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:55,382 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:55,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:55,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:55,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:55,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:55,384 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:55,384 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:55,384 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:55,384 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:55,384 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:55,384 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:55,384 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:55,384 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:55,384 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:55,385 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:55,385 DEBUG [Thread-3278] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:55,385 DEBUG [Thread-3278] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:55,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on fd96548248d41c209e12fca5d7040af5 2023-03-31 16:55:55,387 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing fd96548248d41c209e12fca5d7040af5 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-31 16:55:55,387 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=A 2023-03-31 16:55:55,387 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:55,387 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=B 2023-03-31 16:55:55,388 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:55,393 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=C 2023-03-31 16:55:55,393 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:55,414 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=827 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/A/270b468650f94952a62615d1c8e93190 2023-03-31 16:55:55,420 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:55,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35546 deadline: 1680281815404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:55,421 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:55,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35580 deadline: 1680281815406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:55,436 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:55,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35494 deadline: 1680281815418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:55,437 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:55,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35582 deadline: 1680281815419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:55,437 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:55,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35496 deadline: 1680281815420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:55,463 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=827 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/B/a6327bc1494345408e341be5c6cefe9f 2023-03-31 16:55:55,485 DEBUG [Thread-3278] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:55,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:55,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:55,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:55,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:55,508 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:55,508 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:55,508 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:55,508 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:55,508 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:55,508 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:55,511 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:55,511 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:55,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:55,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:55,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:55,512 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:55,512 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:55,512 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:55,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:55,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:55,512 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:55,529 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:55,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35546 deadline: 1680281815522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:55,536 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:55,536 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:55,536 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:55,536 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:55,536 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:55,538 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:55,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35580 deadline: 1680281815528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:55,547 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:55,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35494 deadline: 1680281815538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:55,547 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:55,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35582 deadline: 1680281815538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:55,547 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:55,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35496 deadline: 1680281815538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:55,549 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:55,549 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:55,550 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:55,550 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. started... 2023-03-31 16:55:55,550 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. as already flushing 2023-03-31 16:55:55,550 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:55,550 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:55,550 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:55,550 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:55,550 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:55,550 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:55,551 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:55,551 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:55,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:55,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:55,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:55,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:55,552 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:55,552 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:55,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:55,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:55,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:55,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:55,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:55,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3807e72d[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:55,553 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:55,553 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:55,554 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:55,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:55,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:55,555 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:55,555 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:55,555 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:55,555 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:55,555 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:55,556 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:55,556 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:55,556 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:55,556 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:55,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:55,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:55,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:55,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:55,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:55,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:55,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:55,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:55,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:55,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:55,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:55,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:55,560 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:55,560 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:55,560 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:55,560 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:55,560 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:55,563 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:55,563 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:55,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:55,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:55,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:55,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:55,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:55,563 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:55,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:55,563 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:55,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:55,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:55,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:55,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:55,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:55,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:55,565 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:55,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:55,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:55,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:55,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:55,567 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:55,567 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:55,567 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:55,567 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:55,567 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:55,567 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:55,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:55,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:55,567 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:55,567 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:55,567 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:55,567 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:55,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:55,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:55,567 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:55,567 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:55,568 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:55,568 DEBUG [Thread-3278] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:55,568 DEBUG [Thread-3278] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:55,668 DEBUG [Thread-3278] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:55,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:55,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:55,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:55,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:55,675 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:55,675 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:55,675 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:55,675 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:55,675 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:55,675 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:55,677 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:55,677 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:55,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:55,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:55,677 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:55,677 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:55,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:55,677 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:55,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:55,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:55,678 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:55,681 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:55,682 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:55,682 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:55,682 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:55,682 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:55,683 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:55,683 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:55,683 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:55,683 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. started... 2023-03-31 16:55:55,683 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. as already flushing 2023-03-31 16:55:55,683 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:55,684 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:55,684 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:55,684 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:55,684 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:55,684 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:55,686 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:55,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:55,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:55,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:55,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:55,688 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:55,688 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:55,688 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:55,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:55,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:55,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:55,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:55,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:55,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16cdb48f[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:55,689 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:55,689 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:55,691 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:55,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:55,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:55,692 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:55,692 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:55,692 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:55,693 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:55,693 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:55,693 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:55,693 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:55,694 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:55,694 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:55,694 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:55,694 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:55,694 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:55,694 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:55,694 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:55,698 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:55,698 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:55,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:55,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:55,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:55,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:55,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:55,699 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:55,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:55,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:55,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:55,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:55,700 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:55,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:55,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:55,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:55,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:55,701 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:55,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:55,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:55,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:55,708 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:55,708 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:55,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:55,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:55,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:55,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:55,710 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:55,710 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:55,710 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:55,710 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:55,710 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:55,710 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:55,710 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:55,710 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:55,710 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:55,710 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:55,710 DEBUG [Thread-3278] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:55,710 DEBUG [Thread-3278] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:55,738 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:55,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35546 deadline: 1680281815732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:55,750 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:55,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35580 deadline: 1680281815740, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:55,757 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:55,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35494 deadline: 1680281815748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:55,757 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:55,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35582 deadline: 1680281815749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:55,758 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:55,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35496 deadline: 1680281815749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:55,811 DEBUG [Thread-3278] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:55,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:55,813 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:55,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:55,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:55,828 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:55,828 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:55,828 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:55,828 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:55,830 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:55,830 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:55,831 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:55,831 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:55,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:55,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:55,832 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:55,832 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:55,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:55,832 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:55,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:55,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:55,832 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:55,843 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:55,843 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:55,843 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:55,844 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:55,844 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:55,852 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:55,852 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:55,868 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:55,868 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. started... 2023-03-31 16:55:55,868 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. as already flushing 2023-03-31 16:55:55,868 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:55,868 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:55,868 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:55,868 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:55,868 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:55,868 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:55,870 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:55,870 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:55,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:55,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:55,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:55,870 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:55,871 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:55,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:55,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:55,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:55,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:55,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:55,873 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:55,873 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5df89d1c[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:55,873 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:55,873 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:55,875 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:55,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:55,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:55,875 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:55,875 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:55,875 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:55,875 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:55,875 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:55,877 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:55,877 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:55,877 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:55,877 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:55,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:55,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:55,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:55,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:55,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:55,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:55,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:55,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:55,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:55,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:55,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:55,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:55,880 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:55,880 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:55,880 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:55,880 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:55,880 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:55,882 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:55,882 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:55,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:55,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:55,883 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:55,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:55,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:55,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:55,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:55,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:55,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:55,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:55,884 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:55,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:55,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:55,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:55,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:55,885 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:55,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:55,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:55,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:55,888 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:55,888 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:55,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:55,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:55,888 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:55,888 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:55,888 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:55,888 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:55,888 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:55,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:55,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:55,888 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:55,889 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:55,889 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:55,889 DEBUG [Thread-3278] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:55,889 DEBUG [Thread-3278] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:55,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:55,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:55,938 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=827 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/C/e5e438dc0e6f4a3bb60c5a64237e7c46 2023-03-31 16:55:55,944 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/A/270b468650f94952a62615d1c8e93190 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/270b468650f94952a62615d1c8e93190 2023-03-31 16:55:55,948 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/270b468650f94952a62615d1c8e93190, entries=200, sequenceid=827, filesize=14.3 K 2023-03-31 16:55:55,949 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/B/a6327bc1494345408e341be5c6cefe9f as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/a6327bc1494345408e341be5c6cefe9f 2023-03-31 16:55:55,957 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/a6327bc1494345408e341be5c6cefe9f, entries=150, sequenceid=827, filesize=11.9 K 2023-03-31 16:55:55,958 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/C/e5e438dc0e6f4a3bb60c5a64237e7c46 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/e5e438dc0e6f4a3bb60c5a64237e7c46 2023-03-31 16:55:55,969 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/e5e438dc0e6f4a3bb60c5a64237e7c46, entries=150, sequenceid=827, filesize=11.9 K 2023-03-31 16:55:55,970 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for fd96548248d41c209e12fca5d7040af5 in 583ms, sequenceid=827, compaction requested=true 2023-03-31 16:55:55,970 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for fd96548248d41c209e12fca5d7040af5: 2023-03-31 16:55:55,970 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:55:55,970 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 16:55:55,970 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 16:55:55,970 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:55:55,970 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:55:55,971 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 51900 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 16:55:55,971 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1912): fd96548248d41c209e12fca5d7040af5/A is initiating minor compaction (all files) 2023-03-31 16:55:55,971 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49460 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 16:55:55,971 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of fd96548248d41c209e12fca5d7040af5/A in TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:55,971 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): fd96548248d41c209e12fca5d7040af5/B is initiating minor compaction (all files) 2023-03-31 16:55:55,971 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/1ca8bc1c03fc4bb8ab992fe817a203c6, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/834a0c767616459084dfac42dd3ad093, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/9b62fccf96fc4ea2bb4f07713b154338, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/270b468650f94952a62615d1c8e93190] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp, totalSize=50.7 K 2023-03-31 16:55:55,971 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fd96548248d41c209e12fca5d7040af5/B in TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:55,971 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/cfb5ffd9bdbb42018e249743caa25b00, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/03352426d4e543c2a58b4f37f24fa868, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/f667690121dc48cf8269b6482f058333, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/a6327bc1494345408e341be5c6cefe9f] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp, totalSize=48.3 K 2023-03-31 16:55:55,972 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 1ca8bc1c03fc4bb8ab992fe817a203c6, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=770, earliestPutTs=1680281752876 2023-03-31 16:55:55,972 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting cfb5ffd9bdbb42018e249743caa25b00, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=770, earliestPutTs=1680281752876 2023-03-31 16:55:55,972 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 834a0c767616459084dfac42dd3ad093, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=790, earliestPutTs=1680281752902 2023-03-31 16:55:55,972 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 03352426d4e543c2a58b4f37f24fa868, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=790, earliestPutTs=1680281752902 2023-03-31 16:55:55,973 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 9b62fccf96fc4ea2bb4f07713b154338, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=807, earliestPutTs=1680281754070 2023-03-31 16:55:55,973 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting f667690121dc48cf8269b6482f058333, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=807, earliestPutTs=1680281754070 2023-03-31 16:55:55,973 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 270b468650f94952a62615d1c8e93190, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=827, earliestPutTs=1680281755238 2023-03-31 16:55:55,974 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting a6327bc1494345408e341be5c6cefe9f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=827, earliestPutTs=1680281755238 2023-03-31 16:55:55,989 DEBUG [Thread-3278] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:55,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:55,992 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:55,994 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] throttle.PressureAwareThroughputController(145): fd96548248d41c209e12fca5d7040af5#A#compaction#503 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:55:56,010 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): fd96548248d41c209e12fca5d7040af5#B#compaction#504 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:55:56,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:56,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:56,015 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:56,015 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:56,015 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:56,015 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:56,015 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:56,016 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:56,017 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:56,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:56,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:56,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:56,017 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:56,018 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:56,018 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:56,018 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:56,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:56,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:56,018 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:56,031 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:56,032 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:56,032 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:56,032 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:56,032 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:56,044 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:56,045 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:56,046 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:56,046 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. started... 2023-03-31 16:55:56,048 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fd96548248d41c209e12fca5d7040af5 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-31 16:55:56,049 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=A 2023-03-31 16:55:56,049 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:56,049 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=B 2023-03-31 16:55:56,049 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:56,049 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=C 2023-03-31 16:55:56,049 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:56,050 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/B/108601ebd8e44dcd801810eb2057018c as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/108601ebd8e44dcd801810eb2057018c 2023-03-31 16:55:56,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on fd96548248d41c209e12fca5d7040af5 2023-03-31 16:55:56,050 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. as already flushing 2023-03-31 16:55:56,052 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/A/8badd180752f4adeb892fe6382b5ebbc as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/8badd180752f4adeb892fe6382b5ebbc 2023-03-31 16:55:56,061 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in fd96548248d41c209e12fca5d7040af5/B of fd96548248d41c209e12fca5d7040af5 into 108601ebd8e44dcd801810eb2057018c(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:55:56,061 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fd96548248d41c209e12fca5d7040af5: 2023-03-31 16:55:56,061 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5., storeName=fd96548248d41c209e12fca5d7040af5/B, priority=12, startTime=1680281755970; duration=0sec 2023-03-31 16:55:56,061 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:55:56,061 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 16:55:56,062 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49460 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 16:55:56,063 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): fd96548248d41c209e12fca5d7040af5/C is initiating minor compaction (all files) 2023-03-31 16:55:56,063 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fd96548248d41c209e12fca5d7040af5/C in TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:56,063 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/ac043966fc7448c0972e50fb65d82bf7, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/5de85fbfe91646239c8619e33f83e550, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/17044945c73e48ec8927e37589b03625, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/e5e438dc0e6f4a3bb60c5a64237e7c46] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp, totalSize=48.3 K 2023-03-31 16:55:56,063 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting ac043966fc7448c0972e50fb65d82bf7, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=770, earliestPutTs=1680281752876 2023-03-31 16:55:56,064 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 5de85fbfe91646239c8619e33f83e550, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=790, earliestPutTs=1680281752902 2023-03-31 16:55:56,064 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 17044945c73e48ec8927e37589b03625, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=807, earliestPutTs=1680281754070 2023-03-31 16:55:56,064 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting e5e438dc0e6f4a3bb60c5a64237e7c46, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=827, earliestPutTs=1680281755238 2023-03-31 16:55:56,073 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in fd96548248d41c209e12fca5d7040af5/A of fd96548248d41c209e12fca5d7040af5 into 8badd180752f4adeb892fe6382b5ebbc(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:55:56,073 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for fd96548248d41c209e12fca5d7040af5: 2023-03-31 16:55:56,073 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5., storeName=fd96548248d41c209e12fca5d7040af5/A, priority=12, startTime=1680281755970; duration=0sec 2023-03-31 16:55:56,074 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:55:56,077 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): fd96548248d41c209e12fca5d7040af5#C#compaction#506 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:55:56,112 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:56,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35496 deadline: 1680281816100, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:56,113 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:56,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35580 deadline: 1680281816099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:56,114 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:56,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35494 deadline: 1680281816102, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:56,114 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:56,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35582 deadline: 1680281816110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:56,120 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=839 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/A/268709b3753c4c19b2398237fec79d6e 2023-03-31 16:55:56,124 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:56,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35546 deadline: 1680281816111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:56,174 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=839 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/B/ae10b461a8634c0db33bef333b6fdec8 2023-03-31 16:55:56,223 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:56,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35580 deadline: 1680281816215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:56,223 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:56,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35496 deadline: 1680281816216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:56,223 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:56,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35494 deadline: 1680281816216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:56,224 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:56,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35582 deadline: 1680281816216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:56,235 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:56,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35546 deadline: 1680281816225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:56,245 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=839 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/C/0d1daaf9d7e54697abb38789b1fd1350 2023-03-31 16:55:56,250 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/A/268709b3753c4c19b2398237fec79d6e as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/268709b3753c4c19b2398237fec79d6e 2023-03-31 16:55:56,254 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/268709b3753c4c19b2398237fec79d6e, entries=150, sequenceid=839, filesize=11.9 K 2023-03-31 16:55:56,255 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/B/ae10b461a8634c0db33bef333b6fdec8 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/ae10b461a8634c0db33bef333b6fdec8 2023-03-31 16:55:56,282 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/ae10b461a8634c0db33bef333b6fdec8, entries=150, sequenceid=839, filesize=11.9 K 2023-03-31 16:55:56,283 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/C/0d1daaf9d7e54697abb38789b1fd1350 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/0d1daaf9d7e54697abb38789b1fd1350 2023-03-31 16:55:56,286 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/0d1daaf9d7e54697abb38789b1fd1350, entries=150, sequenceid=839, filesize=11.9 K 2023-03-31 16:55:56,287 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for fd96548248d41c209e12fca5d7040af5 in 239ms, sequenceid=839, compaction requested=false 2023-03-31 16:55:56,287 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fd96548248d41c209e12fca5d7040af5: 2023-03-31 16:55:56,287 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:56,287 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:56,287 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:56,287 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:56,288 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:56,288 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:56,291 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:56,291 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:56,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:56,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:56,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:56,291 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:56,291 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:56,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:56,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:56,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:56,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:56,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:56,293 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:56,293 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ee941e8[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:56,293 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:56,293 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:56,294 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:56,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:56,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:56,294 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:56,294 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:56,294 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:56,294 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:56,295 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:56,296 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:56,296 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:56,296 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:56,296 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:56,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:56,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:56,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:56,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:56,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:56,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:56,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:56,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:56,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:56,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:56,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:56,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:56,298 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:56,298 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:56,298 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:56,298 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:56,298 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:56,300 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:56,300 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:56,300 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:56,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:56,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:56,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:56,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:56,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:56,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:56,300 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:56,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:56,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:56,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:56,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:56,301 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:56,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:56,304 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:56,304 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:56,304 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:56,304 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:56,304 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:56,304 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:56,304 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:56,304 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:56,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:56,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:56,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:56,304 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:56,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:56,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:56,305 DEBUG [Thread-3278] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:56,305 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:56,305 DEBUG [Thread-3278] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:56,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:56,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:56,305 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:56,405 DEBUG [Thread-3278] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:56,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:56,408 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:56,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:56,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:56,424 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:56,424 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:56,424 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:56,424 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:56,425 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:56,425 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:56,426 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:56,426 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:56,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:56,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:56,442 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:56,442 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:56,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:56,443 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:56,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:56,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:56,444 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:56,444 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:56,444 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:56,444 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:56,444 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:56,454 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:56,454 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:56,455 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:56,455 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. started... 2023-03-31 16:55:56,455 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fd96548248d41c209e12fca5d7040af5 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-31 16:55:56,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on fd96548248d41c209e12fca5d7040af5 2023-03-31 16:55:56,456 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. as already flushing 2023-03-31 16:55:56,457 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680281756098, max timestamp=1680281756452Num uniques 859; 2023-03-31 16:55:56,457 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 16:55:56,457 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680281756098, max timestamp=1680281756452Num uniques 859; 2023-03-31 16:55:56,457 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 16:55:56,457 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680281756098, max timestamp=1680281756452Num uniques 859; 2023-03-31 16:55:56,457 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 16:55:56,460 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=A 2023-03-31 16:55:56,460 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:55:56,460 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=B 2023-03-31 16:55:56,460 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:55:56,460 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=C 2023-03-31 16:55:56,460 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:55:56,503 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680281756475, max timestamp=1680281756500Num uniques 859; 2023-03-31 16:55:56,503 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-31 16:55:56,504 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680281756475, max timestamp=1680281756501Num uniques 859; 2023-03-31 16:55:56,504 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680281756475, max timestamp=1680281756501Num uniques 859; 2023-03-31 16:55:56,504 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-31 16:55:56,504 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-31 16:55:56,513 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=863 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/A/f5c4ef4e600047b9acd2570278f100f4 2023-03-31 16:55:56,545 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/C/fca5c0f726ad48d880420c188dce0c56 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/fca5c0f726ad48d880420c188dce0c56 2023-03-31 16:55:56,552 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in fd96548248d41c209e12fca5d7040af5/C of fd96548248d41c209e12fca5d7040af5 into fca5c0f726ad48d880420c188dce0c56(size=12.7 K), total size for store is 24.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:55:56,552 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fd96548248d41c209e12fca5d7040af5: 2023-03-31 16:55:56,552 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5., storeName=fd96548248d41c209e12fca5d7040af5/C, priority=12, startTime=1680281755970; duration=0sec 2023-03-31 16:55:56,552 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:55:56,558 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680281756519, max timestamp=1680281756554Num uniques 1118; 2023-03-31 16:55:56,558 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=12268, suffix heap size=144240,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=268 2023-03-31 16:55:56,558 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680281756519, max timestamp=1680281756554Num uniques 1118; 2023-03-31 16:55:56,559 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=12268, suffix heap size=144240,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=268 2023-03-31 16:55:56,561 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680281756519, max timestamp=1680281756554Num uniques 1118; 2023-03-31 16:55:56,561 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=12268, suffix heap size=144240,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=268 2023-03-31 16:55:56,607 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=377, cellSize=17262, totalHeapSize=39448, min timestamp=1680281756588, max timestamp=1680281756604Num uniques 1127; 2023-03-31 16:55:56,607 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=17262, suffix heap size=145176,new segment heap size=39448 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=377 2023-03-31 16:55:56,610 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=377, cellSize=17262, totalHeapSize=39448, min timestamp=1680281756588, max timestamp=1680281756604Num uniques 1127; 2023-03-31 16:55:56,610 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=17262, suffix heap size=145176,new segment heap size=39448 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=377 2023-03-31 16:55:56,610 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=377, cellSize=17262, totalHeapSize=39448, min timestamp=1680281756588, max timestamp=1680281756604Num uniques 1127; 2023-03-31 16:55:56,610 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=17262, suffix heap size=145176,new segment heap size=39448 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=377 2023-03-31 16:55:56,682 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:56,684 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1680281756640, max timestamp=1680281756662Num uniques 1236; 2023-03-31 16:55:56,684 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1680281756640, max timestamp=1680281756662Num uniques 1236; 2023-03-31 16:55:56,685 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=56606, new segment data size=10806, suffix heap size=156512,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1236, new segment cells count=236 2023-03-31 16:55:56,684 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1680281756640, max timestamp=1680281756662Num uniques 1236; 2023-03-31 16:55:56,685 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=56606, new segment data size=13096, suffix heap size=156512,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1236, new segment cells count=286 2023-03-31 16:55:56,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35496 deadline: 1680281816642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:56,685 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=56606, new segment data size=10806, suffix heap size=156512,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1236, new segment cells count=236 2023-03-31 16:55:56,746 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=345, cellSize=15800, totalHeapSize=36120, min timestamp=1680281756709, max timestamp=1680281756743Num uniques 1095; 2023-03-31 16:55:56,746 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=15800, suffix heap size=141848,new segment heap size=36120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=345 2023-03-31 16:55:56,746 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=345, cellSize=15800, totalHeapSize=36120, min timestamp=1680281756709, max timestamp=1680281756743Num uniques 1145; 2023-03-31 16:55:56,747 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=15800, suffix heap size=147048,new segment heap size=36120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=345 2023-03-31 16:55:56,749 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1680281756727, max timestamp=1680281756743Num uniques 1095; 2023-03-31 16:55:56,749 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=11220, suffix heap size=141848,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=245 2023-03-31 16:55:56,809 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:56,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35546 deadline: 1680281816793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:56,809 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:56,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 276 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35582 deadline: 1680281816794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:56,810 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:56,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35496 deadline: 1680281816794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:56,811 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:56,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35580 deadline: 1680281816806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:56,821 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:56,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35494 deadline: 1680281816808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:56,913 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:56,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35546 deadline: 1680281816910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:56,913 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:56,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 278 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35582 deadline: 1680281816910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:56,919 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:56,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35496 deadline: 1680281816911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:56,920 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:56,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35580 deadline: 1680281816913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:56,927 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:56,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35494 deadline: 1680281816922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:56,958 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=863 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/B/ac86e89d9098437d89899f7e27cf7d7e 2023-03-31 16:55:56,983 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=863 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/C/774707c3dcbd4d83a8a6c044b07efe1d 2023-03-31 16:55:56,988 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/A/f5c4ef4e600047b9acd2570278f100f4 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/f5c4ef4e600047b9acd2570278f100f4 2023-03-31 16:55:56,991 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/f5c4ef4e600047b9acd2570278f100f4, entries=150, sequenceid=863, filesize=11.9 K 2023-03-31 16:55:56,992 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/B/ac86e89d9098437d89899f7e27cf7d7e as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/ac86e89d9098437d89899f7e27cf7d7e 2023-03-31 16:55:56,995 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/ac86e89d9098437d89899f7e27cf7d7e, entries=150, sequenceid=863, filesize=11.9 K 2023-03-31 16:55:56,996 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/C/774707c3dcbd4d83a8a6c044b07efe1d as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/774707c3dcbd4d83a8a6c044b07efe1d 2023-03-31 16:55:56,999 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/774707c3dcbd4d83a8a6c044b07efe1d, entries=150, sequenceid=863, filesize=11.9 K 2023-03-31 16:55:57,000 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~86.11 KB/88176, currentSize=156.54 KB/160300 for fd96548248d41c209e12fca5d7040af5 in 545ms, sequenceid=863, compaction requested=true 2023-03-31 16:55:57,000 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fd96548248d41c209e12fca5d7040af5: 2023-03-31 16:55:57,000 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:57,000 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:57,000 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:57,000 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,001 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:57,001 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:57,002 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:57,002 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:57,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:57,002 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:57,002 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:57,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:57,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:57,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:57,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:57,004 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:57,004 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29d0e597[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:57,004 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:57,004 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:57,006 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:57,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:57,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:57,007 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:57,007 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:57,007 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:57,007 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,007 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:57,011 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,011 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:57,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:57,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:57,011 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:57,011 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:57,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:57,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:57,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:57,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:57,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:57,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:57,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:57,013 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:57,013 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:57,013 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:57,013 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:57,013 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:57,015 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:57,015 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:57,015 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:57,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:57,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:57,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:57,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:57,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:57,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:57,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:57,016 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:57,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:57,017 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:57,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:57,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:57,022 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:57,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:57,022 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,022 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:57,022 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:57,022 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,022 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:57,022 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:57,022 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:57,022 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:57,022 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:57,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:57,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:57,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:57,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:57,023 DEBUG [Thread-3278] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:57,023 DEBUG [Thread-3278] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:57,023 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:57,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:57,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:57,123 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1680281756789, max timestamp=1680281757120Num uniques 1204; 2023-03-31 16:55:57,123 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1680281756789, max timestamp=1680281757120Num uniques 1204; 2023-03-31 16:55:57,123 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=55140, new segment data size=7050, suffix heap size=153184,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1204, new segment cells count=154 2023-03-31 16:55:57,123 DEBUG [Thread-3278] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:57,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:57,123 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=55140, new segment data size=7050, suffix heap size=153184,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1204, new segment cells count=154 2023-03-31 16:55:57,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on fd96548248d41c209e12fca5d7040af5 2023-03-31 16:55:57,124 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing fd96548248d41c209e12fca5d7040af5 3/3 column families, dataSize=69.33 KB heapSize=190.27 KB 2023-03-31 16:55:57,126 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1680281756789, max timestamp=1680281757120Num uniques 1104; 2023-03-31 16:55:57,126 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-03-31 16:55:57,128 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:57,130 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=A 2023-03-31 16:55:57,130 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:55:57,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:57,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:57,130 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=B 2023-03-31 16:55:57,131 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:57,131 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:57,131 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:57,131 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:57,131 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:55:57,132 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=C 2023-03-31 16:55:57,132 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:55:57,140 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:57,141 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:57,147 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:57,147 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:57,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:57,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:57,148 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,148 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:57,148 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:57,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:57,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:57,149 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:57,154 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:57,154 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:57,154 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:57,154 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:57,154 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:57,156 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:57,156 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:57,156 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:57,156 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. started... 2023-03-31 16:55:57,156 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. as already flushing 2023-03-31 16:55:57,156 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:57,156 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:57,156 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:57,156 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,163 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:57,164 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:57,165 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:57,166 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:57,166 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:57,167 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:57,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:57,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:57,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:57,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:57,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:57,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:57,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@524255ee[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:57,170 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:57,170 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:57,171 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:57,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:57,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:57,171 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:57,171 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:57,171 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:57,172 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,172 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:57,173 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,173 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:57,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:57,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:57,173 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:57,173 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:57,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:57,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:57,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:57,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:57,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:57,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:57,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:57,175 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:57,175 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:57,175 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:57,175 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:57,175 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:57,181 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:57,181 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:57,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:57,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:57,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:57,181 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:57,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:57,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:57,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:57,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:57,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:57,182 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:57,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:57,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:57,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:57,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:57,186 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:57,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,188 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,188 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:57,188 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:57,188 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:57,188 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,188 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:57,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:57,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:57,188 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:57,188 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:57,188 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:57,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:57,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:57,188 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:57,188 DEBUG [Thread-3278] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:57,188 DEBUG [Thread-3278] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:57,188 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:57,189 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:57,194 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680281757177, max timestamp=1680281757193Num uniques 859; 2023-03-31 16:55:57,194 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-31 16:55:57,195 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680281757177, max timestamp=1680281757193Num uniques 859; 2023-03-31 16:55:57,195 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680281757177, max timestamp=1680281757193Num uniques 859; 2023-03-31 16:55:57,195 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-31 16:55:57,195 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-31 16:55:57,233 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:57,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35496 deadline: 1680281817224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,235 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=368, cellSize=16848, totalHeapSize=38512, min timestamp=1680281757214, max timestamp=1680281757232Num uniques 1118; 2023-03-31 16:55:57,235 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=16848, suffix heap size=144240,new segment heap size=38512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=368 2023-03-31 16:55:57,235 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=368, cellSize=16848, totalHeapSize=38512, min timestamp=1680281757214, max timestamp=1680281757232Num uniques 1118; 2023-03-31 16:55:57,235 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=16848, suffix heap size=144240,new segment heap size=38512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=368 2023-03-31 16:55:57,235 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=368, cellSize=16848, totalHeapSize=38512, min timestamp=1680281757214, max timestamp=1680281757232Num uniques 1118; 2023-03-31 16:55:57,235 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=16848, suffix heap size=144240,new segment heap size=38512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=368 2023-03-31 16:55:57,286 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:57,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 281 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35494 deadline: 1680281817276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,286 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:57,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 280 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35580 deadline: 1680281817277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,289 DEBUG [Thread-3278] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:57,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:57,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:57,291 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:57,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35546 deadline: 1680281817283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:57,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:57,296 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:57,296 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:57,296 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:57,296 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:57,296 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:57,297 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:57,297 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:57,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 292 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35582 deadline: 1680281817284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,298 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:57,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:57,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:57,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:57,299 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,299 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:57,299 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:57,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:57,299 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:57,299 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:57,304 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:57,304 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:57,304 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:57,304 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:57,304 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:57,313 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:57,313 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:57,313 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:57,313 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. started... 2023-03-31 16:55:57,313 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. as already flushing 2023-03-31 16:55:57,313 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:57,313 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:57,313 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:57,313 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,314 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:57,314 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:57,315 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,315 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:57,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:57,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:57,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:57,316 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:57,316 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:57,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:57,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:57,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:57,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:57,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4bcb32c7[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:57,319 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:57,319 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:57,321 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:57,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:57,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:57,321 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:57,322 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:57,322 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:57,322 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,322 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:57,323 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:57,323 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:57,323 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:57,323 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:57,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:57,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:57,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:57,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:57,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:57,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:57,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:57,328 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:57,328 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:57,328 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:57,328 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:57,328 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:57,328 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:57,330 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:57,330 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:57,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:57,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:57,330 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:57,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:57,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:57,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:57,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:57,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:57,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:57,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:57,332 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:57,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:57,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:57,333 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:57,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:57,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,337 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:57,337 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:57,337 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:57,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:57,337 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:57,338 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:57,338 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,338 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:57,338 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:57,338 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:57,338 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:57,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:57,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:57,338 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:57,338 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:57,338 DEBUG [Thread-3278] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:57,338 DEBUG [Thread-3278] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:57,338 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:57,349 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:57,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35496 deadline: 1680281817337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,396 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:57,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 282 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35580 deadline: 1680281817387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,397 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:57,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 283 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35494 deadline: 1680281817388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,397 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:57,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35546 deadline: 1680281817392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,407 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:57,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 294 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35582 deadline: 1680281817399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,438 DEBUG [Thread-3278] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:57,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:57,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:57,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:57,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:57,453 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:57,453 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:57,453 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:57,453 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:57,455 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:57,455 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:57,457 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,457 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:57,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:57,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:57,457 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,457 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:57,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:57,457 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:57,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:57,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:57,458 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:57,460 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:57,460 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:57,460 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:57,460 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:57,460 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:57,464 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:57,464 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:57,464 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:57,464 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. started... 2023-03-31 16:55:57,464 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. as already flushing 2023-03-31 16:55:57,464 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:57,464 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:57,464 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:57,464 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,464 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:57,464 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:57,466 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:57,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:57,466 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:57,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:57,467 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:57,467 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:57,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:57,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:57,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:57,468 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:57,468 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2fc9cc57[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:57,468 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:57,468 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:57,469 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:57,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:57,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:57,470 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:57,470 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:57,470 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:57,472 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:57,472 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:57,472 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:57,472 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,472 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:57,472 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:57,472 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:57,472 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:57,472 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:57,472 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:57,472 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:57,474 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:57,475 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:57,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:57,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:57,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:57,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:57,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:57,475 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:57,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:57,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:57,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:57,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:57,476 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:57,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:57,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:57,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,477 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:57,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:57,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,480 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:57,480 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:57,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:57,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:57,480 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,480 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:57,480 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:57,480 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,480 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:57,480 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:57,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:57,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:57,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:57,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:57,480 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:57,481 DEBUG [Thread-3278] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:57,481 DEBUG [Thread-3278] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:57,480 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:57,553 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:57,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35496 deadline: 1680281817550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,573 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=972 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/A/cf6deddab5a049de9c38d7e4fbedab28 2023-03-31 16:55:57,581 DEBUG [Thread-3278] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:57,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:57,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:57,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:57,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:57,584 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:57,584 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:57,584 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:57,584 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:57,585 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:57,585 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:57,588 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:57,588 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:57,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:57,588 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,588 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:57,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:57,589 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:57,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:57,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:57,589 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:57,602 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:57,602 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:57,602 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:57,602 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:57,602 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:57,604 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:57,604 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:57,604 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=972 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/B/4a56398418664a7e93addb403d9e0a5f 2023-03-31 16:55:57,604 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:57,605 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. started... 2023-03-31 16:55:57,605 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. as already flushing 2023-03-31 16:55:57,605 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:57,605 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:57,605 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:57,606 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,606 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:57,606 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:57,607 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:57,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 284 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35580 deadline: 1680281817599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,607 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:57,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35546 deadline: 1680281817599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,607 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:57,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 285 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35494 deadline: 1680281817600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,613 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,613 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:57,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:57,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:57,614 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:57,614 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:57,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:57,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:57,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:57,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:57,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:57,616 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11b6864[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:57,616 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:57,616 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:57,617 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:57,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:57,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:57,617 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:57,617 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:57,617 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:57,617 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,618 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:57,619 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:57,619 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:57,619 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:57,619 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:57,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:57,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:57,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:57,620 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:57,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 296 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35582 deadline: 1680281817610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:57,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:57,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:57,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:57,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:57,623 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:57,623 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:57,623 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:57,623 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:57,623 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:57,625 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:57,625 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:57,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:57,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:57,625 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,626 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,629 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:57,629 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:57,629 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:57,629 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:57,630 DEBUG [Thread-3278] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:57,630 DEBUG [Thread-3278] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:57,625 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:57,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:57,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:57,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:57,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:57,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:57,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:57,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:57,636 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,637 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:57,637 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:57,637 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,637 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:57,637 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:57,637 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:57,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:57,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:57,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:57,640 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:57,651 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=972 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/C/2864308f27f545a09962fb08fe7eca01 2023-03-31 16:55:57,654 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/A/cf6deddab5a049de9c38d7e4fbedab28 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/cf6deddab5a049de9c38d7e4fbedab28 2023-03-31 16:55:57,659 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/cf6deddab5a049de9c38d7e4fbedab28, entries=150, sequenceid=972, filesize=11.9 K 2023-03-31 16:55:57,659 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/B/4a56398418664a7e93addb403d9e0a5f as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/4a56398418664a7e93addb403d9e0a5f 2023-03-31 16:55:57,673 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/4a56398418664a7e93addb403d9e0a5f, entries=150, sequenceid=972, filesize=11.9 K 2023-03-31 16:55:57,674 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/C/2864308f27f545a09962fb08fe7eca01 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/2864308f27f545a09962fb08fe7eca01 2023-03-31 16:55:57,677 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/2864308f27f545a09962fb08fe7eca01, entries=150, sequenceid=972, filesize=11.9 K 2023-03-31 16:55:57,678 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~86.58 KB/88656, currentSize=161.02 KB/164880 for fd96548248d41c209e12fca5d7040af5 in 554ms, sequenceid=972, compaction requested=true 2023-03-31 16:55:57,678 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for fd96548248d41c209e12fca5d7040af5: 2023-03-31 16:55:57,678 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:55:57,678 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 16:55:57,678 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 16:55:57,678 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:55:57,678 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:55:57,679 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49596 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 16:55:57,679 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49596 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 16:55:57,679 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): fd96548248d41c209e12fca5d7040af5/B is initiating minor compaction (all files) 2023-03-31 16:55:57,679 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1912): fd96548248d41c209e12fca5d7040af5/A is initiating minor compaction (all files) 2023-03-31 16:55:57,679 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fd96548248d41c209e12fca5d7040af5/B in TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:57,679 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of fd96548248d41c209e12fca5d7040af5/A in TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:57,679 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/108601ebd8e44dcd801810eb2057018c, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/ae10b461a8634c0db33bef333b6fdec8, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/ac86e89d9098437d89899f7e27cf7d7e, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/4a56398418664a7e93addb403d9e0a5f] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp, totalSize=48.4 K 2023-03-31 16:55:57,679 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/8badd180752f4adeb892fe6382b5ebbc, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/268709b3753c4c19b2398237fec79d6e, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/f5c4ef4e600047b9acd2570278f100f4, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/cf6deddab5a049de9c38d7e4fbedab28] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp, totalSize=48.4 K 2023-03-31 16:55:57,680 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 108601ebd8e44dcd801810eb2057018c, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=827, earliestPutTs=1680281755238 2023-03-31 16:55:57,680 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 8badd180752f4adeb892fe6382b5ebbc, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=827, earliestPutTs=1680281755238 2023-03-31 16:55:57,680 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting ae10b461a8634c0db33bef333b6fdec8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=839, earliestPutTs=1680281755404 2023-03-31 16:55:57,680 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 268709b3753c4c19b2398237fec79d6e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=839, earliestPutTs=1680281755404 2023-03-31 16:55:57,680 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting ac86e89d9098437d89899f7e27cf7d7e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=863, earliestPutTs=1680281756098 2023-03-31 16:55:57,681 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting f5c4ef4e600047b9acd2570278f100f4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=863, earliestPutTs=1680281756098 2023-03-31 16:55:57,681 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 4a56398418664a7e93addb403d9e0a5f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=972, earliestPutTs=1680281756805 2023-03-31 16:55:57,681 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting cf6deddab5a049de9c38d7e4fbedab28, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=972, earliestPutTs=1680281756805 2023-03-31 16:55:57,693 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): fd96548248d41c209e12fca5d7040af5#B#compaction#515 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:55:57,700 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] throttle.PressureAwareThroughputController(145): fd96548248d41c209e12fca5d7040af5#A#compaction#516 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:55:57,713 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/B/68d5852e59c7436281da1157e0363497 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/68d5852e59c7436281da1157e0363497 2023-03-31 16:55:57,715 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/A/d95c5e11761941ad996f1190345a0f82 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/d95c5e11761941ad996f1190345a0f82 2023-03-31 16:55:57,719 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in fd96548248d41c209e12fca5d7040af5/B of fd96548248d41c209e12fca5d7040af5 into 68d5852e59c7436281da1157e0363497(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:55:57,719 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fd96548248d41c209e12fca5d7040af5: 2023-03-31 16:55:57,719 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5., storeName=fd96548248d41c209e12fca5d7040af5/B, priority=12, startTime=1680281757678; duration=0sec 2023-03-31 16:55:57,719 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:55:57,719 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 16:55:57,721 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49596 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 16:55:57,721 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): fd96548248d41c209e12fca5d7040af5/C is initiating minor compaction (all files) 2023-03-31 16:55:57,721 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fd96548248d41c209e12fca5d7040af5/C in TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:57,721 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/fca5c0f726ad48d880420c188dce0c56, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/0d1daaf9d7e54697abb38789b1fd1350, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/774707c3dcbd4d83a8a6c044b07efe1d, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/2864308f27f545a09962fb08fe7eca01] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp, totalSize=48.4 K 2023-03-31 16:55:57,723 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting fca5c0f726ad48d880420c188dce0c56, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=827, earliestPutTs=1680281755238 2023-03-31 16:55:57,724 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 0d1daaf9d7e54697abb38789b1fd1350, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=839, earliestPutTs=1680281755404 2023-03-31 16:55:57,724 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 774707c3dcbd4d83a8a6c044b07efe1d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=863, earliestPutTs=1680281756098 2023-03-31 16:55:57,724 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 2864308f27f545a09962fb08fe7eca01, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=972, earliestPutTs=1680281756805 2023-03-31 16:55:57,725 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in fd96548248d41c209e12fca5d7040af5/A of fd96548248d41c209e12fca5d7040af5 into d95c5e11761941ad996f1190345a0f82(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:55:57,725 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for fd96548248d41c209e12fca5d7040af5: 2023-03-31 16:55:57,725 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5., storeName=fd96548248d41c209e12fca5d7040af5/A, priority=12, startTime=1680281757678; duration=0sec 2023-03-31 16:55:57,725 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:55:57,730 DEBUG [Thread-3278] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:57,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:57,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:57,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:57,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:57,740 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:57,740 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:57,740 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:57,740 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:57,741 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:57,741 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:57,742 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): fd96548248d41c209e12fca5d7040af5#C#compaction#517 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:55:57,742 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:57,742 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:57,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:57,743 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,743 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:57,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:57,743 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:57,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:57,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:57,745 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:57,748 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:57,748 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:57,748 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:57,748 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:57,748 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:57,748 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:57,748 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:57,748 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:57,748 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. started... 2023-03-31 16:55:57,749 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fd96548248d41c209e12fca5d7040af5 3/3 column families, dataSize=161.02 KB heapSize=445.08 KB 2023-03-31 16:55:57,749 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=A 2023-03-31 16:55:57,749 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:55:57,749 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=B 2023-03-31 16:55:57,749 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:55:57,749 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=C 2023-03-31 16:55:57,749 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:55:57,780 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/C/83f492e3c71c4a7cb0e07753a923555f as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/83f492e3c71c4a7cb0e07753a923555f 2023-03-31 16:55:57,785 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in fd96548248d41c209e12fca5d7040af5/C of fd96548248d41c209e12fca5d7040af5 into 83f492e3c71c4a7cb0e07753a923555f(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:55:57,786 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fd96548248d41c209e12fca5d7040af5: 2023-03-31 16:55:57,786 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5., storeName=fd96548248d41c209e12fca5d7040af5/C, priority=12, startTime=1680281757678; duration=0sec 2023-03-31 16:55:57,786 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:55:57,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on fd96548248d41c209e12fca5d7040af5 2023-03-31 16:55:57,863 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. as already flushing 2023-03-31 16:55:57,883 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:57,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 281 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35496 deadline: 1680281817879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,913 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:57,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 286 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35580 deadline: 1680281817909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,913 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:57,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35546 deadline: 1680281817910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,913 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:57,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 287 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35494 deadline: 1680281817910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,926 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:57,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 298 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35582 deadline: 1680281817922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:57,989 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:57,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 283 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35496 deadline: 1680281817984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,193 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:58,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 285 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35496 deadline: 1680281818191, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,202 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=1028 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/A/014712c3f1bc49e3bf4e68b0c826ad38 2023-03-31 16:55:58,222 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=1028 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/B/cea9a4e7da7847f0af7d82800c562d22 2023-03-31 16:55:58,260 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=1028 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/C/dd6edc3e7b294e45a993e1b8ae74bf6a 2023-03-31 16:55:58,268 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/A/014712c3f1bc49e3bf4e68b0c826ad38 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/014712c3f1bc49e3bf4e68b0c826ad38 2023-03-31 16:55:58,275 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/014712c3f1bc49e3bf4e68b0c826ad38, entries=150, sequenceid=1028, filesize=11.9 K 2023-03-31 16:55:58,275 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/B/cea9a4e7da7847f0af7d82800c562d22 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/cea9a4e7da7847f0af7d82800c562d22 2023-03-31 16:55:58,285 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/cea9a4e7da7847f0af7d82800c562d22, entries=150, sequenceid=1028, filesize=11.9 K 2023-03-31 16:55:58,286 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/C/dd6edc3e7b294e45a993e1b8ae74bf6a as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/dd6edc3e7b294e45a993e1b8ae74bf6a 2023-03-31 16:55:58,295 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/dd6edc3e7b294e45a993e1b8ae74bf6a, entries=150, sequenceid=1028, filesize=11.9 K 2023-03-31 16:55:58,296 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~161.02 KB/164880, heapSize ~445.03 KB/455712, currentSize=26.84 KB/27480 for fd96548248d41c209e12fca5d7040af5 in 547ms, sequenceid=1028, compaction requested=false 2023-03-31 16:55:58,296 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fd96548248d41c209e12fca5d7040af5: 2023-03-31 16:55:58,296 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:58,296 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:58,296 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:58,296 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,296 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:58,296 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:58,298 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,298 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:58,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:58,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:58,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:58,298 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:58,299 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:58,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:58,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:58,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:58,302 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:58,302 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41b6a08c[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:58,302 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:58,302 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:58,303 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:58,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:58,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:58,304 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:58,304 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:58,304 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:58,306 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:58,306 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:58,306 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:58,306 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,307 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:58,307 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:58,308 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:58,308 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:58,308 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:58,308 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:58,308 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:58,313 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:58,313 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:58,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:58,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:58,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:58,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:58,313 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:58,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:58,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:58,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:58,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:58,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:58,316 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:58,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:58,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:58,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,319 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:58,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:58,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,324 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,324 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:58,324 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:58,324 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,324 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:58,324 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:58,324 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:58,325 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:58,324 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:58,325 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:58,324 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:58,325 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:58,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:58,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:58,325 DEBUG [Thread-3278] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:58,325 DEBUG [Thread-3278] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:58,325 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:58,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:58,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:58,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on fd96548248d41c209e12fca5d7040af5 2023-03-31 16:55:58,421 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing fd96548248d41c209e12fca5d7040af5 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-31 16:55:58,421 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=A 2023-03-31 16:55:58,421 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:58,422 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=B 2023-03-31 16:55:58,422 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:58,422 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=C 2023-03-31 16:55:58,422 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:55:58,425 DEBUG [Thread-3278] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:58,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:58,427 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:58,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:58,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:58,432 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:58,432 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:58,432 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:58,432 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:58,432 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:58,433 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:58,434 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,434 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:58,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:58,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:58,434 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,434 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:58,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:58,435 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:58,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:58,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:58,435 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:58,440 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:58,440 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:58,440 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:58,440 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:58,440 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:58,447 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:58,447 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:58,447 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:58,447 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. started... 2023-03-31 16:55:58,447 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. as already flushing 2023-03-31 16:55:58,447 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:58,447 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:58,447 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:58,447 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,447 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:58,448 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:58,456 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,456 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:58,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:58,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:58,456 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:58,457 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:58,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:58,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:58,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:58,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:58,458 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:58,458 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5379ff77[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:58,458 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:58,458 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:58,459 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:58,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:58,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:58,460 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:58,460 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:58,460 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:58,460 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,460 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:58,461 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,461 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:58,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:58,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:58,461 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:58,462 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:58,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:58,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:58,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:58,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:58,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:58,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:58,464 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:58,464 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:58,464 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:58,464 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:58,464 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:58,464 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:58,466 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:58,466 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:58,466 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:58,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:58,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:58,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:58,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:58,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:58,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:58,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:58,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:58,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:58,467 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:58,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:58,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:58,468 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:58,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:58,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,472 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:58,472 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:58,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:58,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:58,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:58,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:58,473 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,473 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:58,473 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:58,473 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,473 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:58,473 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:58,473 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:58,474 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:58,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:58,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:58,475 DEBUG [Thread-3278] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:58,475 DEBUG [Thread-3278] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:58,476 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:58,478 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680281758436, max timestamp=1680281758476Num uniques 859; 2023-03-31 16:55:58,478 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680281758436, max timestamp=1680281758476Num uniques 859; 2023-03-31 16:55:58,479 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 16:55:58,478 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 16:55:58,480 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680281758436, max timestamp=1680281758476Num uniques 859; 2023-03-31 16:55:58,480 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 16:55:58,509 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:58,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 298 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35494 deadline: 1680281818500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,510 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:58,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 287 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35546 deadline: 1680281818501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,517 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:58,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35496 deadline: 1680281818509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,518 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:58,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 307 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35582 deadline: 1680281818509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,520 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:58,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 297 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35580 deadline: 1680281818510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,575 DEBUG [Thread-3278] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:58,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:58,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:58,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:58,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:58,578 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:58,578 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:58,578 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:58,578 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:58,579 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:58,579 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:58,580 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:58,580 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:58,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:58,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:58,581 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,581 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:58,581 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:58,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:58,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:58,581 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:58,581 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:58,581 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:58,582 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:58,582 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:58,582 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:58,582 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:58,582 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:58,582 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:58,582 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. started... 2023-03-31 16:55:58,582 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. as already flushing 2023-03-31 16:55:58,582 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:58,582 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:58,582 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:58,582 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,582 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:58,582 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:58,585 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,585 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:58,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:58,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:58,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:58,587 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:58,587 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:58,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:58,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:58,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:58,588 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:58,588 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32b33a3b[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:58,588 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:58,588 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:58,593 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:58,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:58,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:58,594 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,594 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:58,594 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:58,594 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:58,594 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:58,595 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:58,595 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:58,595 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:58,595 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:58,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:58,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:58,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:58,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:58,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:58,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:58,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:58,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:58,598 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:58,598 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:58,598 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:58,599 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:58,599 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:58,600 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:58,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:58,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:58,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:58,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:58,601 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:58,601 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:58,602 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:58,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:58,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:58,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:58,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:58,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:58,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:58,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,603 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:58,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:58,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:58,607 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,607 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:58,607 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:58,607 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,607 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:58,608 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:58,608 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:58,608 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:58,608 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:58,608 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:58,608 DEBUG [Thread-3278] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:58,608 DEBUG [Thread-3278] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:58,611 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:58,611 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:58,611 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:58,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:58,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:58,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:58,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:58,613 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:58,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 300 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35494 deadline: 1680281818610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,620 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:58,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 289 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35546 deadline: 1680281818611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,625 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:58,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 309 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35582 deadline: 1680281818619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,626 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:58,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35496 deadline: 1680281818620, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,633 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:58,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 299 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35580 deadline: 1680281818623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,708 DEBUG [Thread-3278] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:58,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:58,710 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:58,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:58,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:58,711 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:58,712 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:58,712 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:58,712 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:58,712 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:58,712 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:58,713 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:58,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:58,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:58,713 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,714 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,714 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:58,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:58,715 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:58,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:58,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:58,716 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:58,716 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:58,716 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:58,716 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:58,716 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:58,716 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:58,716 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:58,716 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:58,717 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:58,717 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. started... 2023-03-31 16:55:58,717 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. as already flushing 2023-03-31 16:55:58,717 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:58,717 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:58,717 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:58,717 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,717 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:58,717 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:58,718 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,719 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:58,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:58,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:58,719 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:58,719 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:58,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:58,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:58,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:58,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:58,721 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:58,721 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68313c41[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:58,721 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:58,721 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:58,723 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,723 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:58,723 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:58,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:58,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:58,724 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:58,724 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:58,724 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:58,726 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,726 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:58,726 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:58,727 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:58,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:58,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:58,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:58,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:58,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:58,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:58,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:58,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:58,730 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:58,730 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:58,730 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:58,730 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:58,730 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:58,730 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:58,732 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:58,732 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:58,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:58,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:58,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:58,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:58,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:58,732 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:58,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:58,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:58,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:58,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:58,733 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:58,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:58,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:58,736 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:58,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:58,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,738 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,738 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:58,738 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:58,738 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,738 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:58,738 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:58,738 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:58,738 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:58,738 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:58,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:58,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:58,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:58,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:58,739 DEBUG [Thread-3278] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:58,739 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:58,738 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:58,739 DEBUG [Thread-3278] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:58,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:58,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:58,822 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:58,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 302 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35494 deadline: 1680281818815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,835 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:58,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 291 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35546 deadline: 1680281818828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,836 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:58,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 311 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35582 deadline: 1680281818828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,836 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:58,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 292 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35496 deadline: 1680281818828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,839 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1040 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/A/8fd2c9059f59463eac1393cbebd2509f 2023-03-31 16:55:58,841 DEBUG [Thread-3278] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:58,842 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:58,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:58,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 301 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35580 deadline: 1680281818834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:58,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:58,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:58,849 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:58,850 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:58,850 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:58,850 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:58,851 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:58,851 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:58,853 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,853 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:58,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:58,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:58,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:58,853 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,854 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:58,854 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:58,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:58,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:58,854 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:58,856 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:58,856 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:58,856 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:58,856 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:58,856 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:58,868 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:58,868 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:58,868 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1040 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/B/2e378fdcfd9846b6af7146900eccaede 2023-03-31 16:55:58,869 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:58,869 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. started... 2023-03-31 16:55:58,869 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. as already flushing 2023-03-31 16:55:58,869 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:58,869 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:58,869 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:58,869 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,869 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:58,870 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:58,872 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:58,872 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:58,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:58,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:58,873 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:58,873 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:58,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:58,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:58,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:58,874 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:58,875 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c77430b[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:58,875 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:58,875 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:58,876 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:58,876 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,876 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:58,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:58,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:58,876 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:58,876 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:58,877 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:58,878 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:58,878 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,878 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:58,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:58,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:58,878 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:58,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:58,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:58,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:58,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:58,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:58,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:58,885 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:58,885 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:58,885 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:58,885 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:58,885 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:58,885 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:58,887 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:58,887 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:58,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:58,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:58,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:58,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:58,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:58,887 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:58,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:58,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:58,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:58,888 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:58,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:58,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:58,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:58,889 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:58,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:58,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,891 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1040 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/C/c87d16ea699f4075ab53b0a18f2d9821 2023-03-31 16:55:58,893 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:58,894 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:58,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:58,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:58,894 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:58,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:58,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:58,894 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:55:58,894 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,894 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:58,895 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:58,895 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:58,895 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:58,895 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:58,895 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:58,894 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:58,895 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:58,896 DEBUG [Thread-3278] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:58,896 DEBUG [Thread-3278] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:58,896 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/A/8fd2c9059f59463eac1393cbebd2509f as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/8fd2c9059f59463eac1393cbebd2509f 2023-03-31 16:55:58,905 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/8fd2c9059f59463eac1393cbebd2509f, entries=200, sequenceid=1040, filesize=14.3 K 2023-03-31 16:55:58,906 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/B/2e378fdcfd9846b6af7146900eccaede as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/2e378fdcfd9846b6af7146900eccaede 2023-03-31 16:55:58,911 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/2e378fdcfd9846b6af7146900eccaede, entries=150, sequenceid=1040, filesize=11.9 K 2023-03-31 16:55:58,911 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/C/c87d16ea699f4075ab53b0a18f2d9821 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/c87d16ea699f4075ab53b0a18f2d9821 2023-03-31 16:55:58,915 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/c87d16ea699f4075ab53b0a18f2d9821, entries=150, sequenceid=1040, filesize=11.9 K 2023-03-31 16:55:58,915 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for fd96548248d41c209e12fca5d7040af5 in 494ms, sequenceid=1040, compaction requested=true 2023-03-31 16:55:58,915 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for fd96548248d41c209e12fca5d7040af5: 2023-03-31 16:55:58,916 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:55:58,916 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 16:55:58,916 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 16:55:58,916 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:55:58,916 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:55:58,917 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37546 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 16:55:58,917 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): fd96548248d41c209e12fca5d7040af5/B is initiating minor compaction (all files) 2023-03-31 16:55:58,917 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fd96548248d41c209e12fca5d7040af5/B in TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:58,917 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/68d5852e59c7436281da1157e0363497, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/cea9a4e7da7847f0af7d82800c562d22, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/2e378fdcfd9846b6af7146900eccaede] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp, totalSize=36.7 K 2023-03-31 16:55:58,917 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39986 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 16:55:58,917 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1912): fd96548248d41c209e12fca5d7040af5/A is initiating minor compaction (all files) 2023-03-31 16:55:58,917 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of fd96548248d41c209e12fca5d7040af5/A in TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:58,917 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/d95c5e11761941ad996f1190345a0f82, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/014712c3f1bc49e3bf4e68b0c826ad38, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/8fd2c9059f59463eac1393cbebd2509f] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp, totalSize=39.0 K 2023-03-31 16:55:58,918 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 68d5852e59c7436281da1157e0363497, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=972, earliestPutTs=1680281756805 2023-03-31 16:55:58,919 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting cea9a4e7da7847f0af7d82800c562d22, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1028, earliestPutTs=1680281757275 2023-03-31 16:55:58,919 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting d95c5e11761941ad996f1190345a0f82, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=972, earliestPutTs=1680281756805 2023-03-31 16:55:58,920 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 2e378fdcfd9846b6af7146900eccaede, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1040, earliestPutTs=1680281757862 2023-03-31 16:55:58,920 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 014712c3f1bc49e3bf4e68b0c826ad38, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1028, earliestPutTs=1680281757275 2023-03-31 16:55:58,920 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 8fd2c9059f59463eac1393cbebd2509f, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=1040, earliestPutTs=1680281757862 2023-03-31 16:55:58,929 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): fd96548248d41c209e12fca5d7040af5#B#compaction#524 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:55:58,930 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] throttle.PressureAwareThroughputController(145): fd96548248d41c209e12fca5d7040af5#A#compaction#525 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:55:58,952 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/B/2dad9ef02ddc44c1a45bc45f2f5442dd as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/2dad9ef02ddc44c1a45bc45f2f5442dd 2023-03-31 16:55:58,958 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fd96548248d41c209e12fca5d7040af5/B of fd96548248d41c209e12fca5d7040af5 into 2dad9ef02ddc44c1a45bc45f2f5442dd(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:55:58,958 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fd96548248d41c209e12fca5d7040af5: 2023-03-31 16:55:58,959 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5., storeName=fd96548248d41c209e12fca5d7040af5/B, priority=13, startTime=1680281758916; duration=0sec 2023-03-31 16:55:58,959 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:55:58,959 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 16:55:58,960 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37546 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 16:55:58,960 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): fd96548248d41c209e12fca5d7040af5/C is initiating minor compaction (all files) 2023-03-31 16:55:58,960 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fd96548248d41c209e12fca5d7040af5/C in TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:58,960 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/83f492e3c71c4a7cb0e07753a923555f, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/dd6edc3e7b294e45a993e1b8ae74bf6a, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/c87d16ea699f4075ab53b0a18f2d9821] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp, totalSize=36.7 K 2023-03-31 16:55:58,961 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 83f492e3c71c4a7cb0e07753a923555f, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=972, earliestPutTs=1680281756805 2023-03-31 16:55:58,961 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting dd6edc3e7b294e45a993e1b8ae74bf6a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1028, earliestPutTs=1680281757275 2023-03-31 16:55:58,961 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting c87d16ea699f4075ab53b0a18f2d9821, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1040, earliestPutTs=1680281757862 2023-03-31 16:55:58,968 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): fd96548248d41c209e12fca5d7040af5#C#compaction#526 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:55:58,988 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/C/033a6a82187c4565a948c7681bdd7672 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/033a6a82187c4565a948c7681bdd7672 2023-03-31 16:55:58,996 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fd96548248d41c209e12fca5d7040af5/C of fd96548248d41c209e12fca5d7040af5 into 033a6a82187c4565a948c7681bdd7672(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:55:58,996 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fd96548248d41c209e12fca5d7040af5: 2023-03-31 16:55:58,996 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5., storeName=fd96548248d41c209e12fca5d7040af5/C, priority=13, startTime=1680281758916; duration=0sec 2023-03-31 16:55:58,996 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:55:58,996 DEBUG [Thread-3278] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:55:58,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:55:58,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:55:59,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:55:59,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:55:59,007 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:55:59,007 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:59,007 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:55:59,007 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:55:59,008 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:59,008 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:59,010 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:59,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:59,010 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:59,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:59,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:59,011 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:59,011 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:55:59,011 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:59,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:55:59,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:59,012 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:55:59,022 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:55:59,022 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:55:59,022 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:55:59,023 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:55:59,023 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:55:59,024 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:55:59,024 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:55:59,024 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:59,024 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. started... 2023-03-31 16:55:59,024 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fd96548248d41c209e12fca5d7040af5 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-03-31 16:55:59,024 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=A 2023-03-31 16:55:59,025 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:55:59,025 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=B 2023-03-31 16:55:59,025 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:55:59,025 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=C 2023-03-31 16:55:59,025 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:55:59,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on fd96548248d41c209e12fca5d7040af5 2023-03-31 16:55:59,130 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. as already flushing 2023-03-31 16:55:59,152 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:59,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 294 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35546 deadline: 1680281819145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:59,153 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:59,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 307 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35494 deadline: 1680281819146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:59,154 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:59,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 303 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35580 deadline: 1680281819146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:59,154 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:59,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 315 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35582 deadline: 1680281819149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:59,161 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:59,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 296 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35496 deadline: 1680281819153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:59,262 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:59,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 296 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35546 deadline: 1680281819254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:59,262 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:59,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 309 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35494 deadline: 1680281819254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:59,263 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:59,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 317 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35582 deadline: 1680281819255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:59,269 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:59,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 298 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35496 deadline: 1680281819262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:59,348 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/A/700a56804c0544f0a918699941cd13f4 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/700a56804c0544f0a918699941cd13f4 2023-03-31 16:55:59,357 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fd96548248d41c209e12fca5d7040af5/A of fd96548248d41c209e12fca5d7040af5 into 700a56804c0544f0a918699941cd13f4(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:55:59,357 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for fd96548248d41c209e12fca5d7040af5: 2023-03-31 16:55:59,357 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5., storeName=fd96548248d41c209e12fca5d7040af5/A, priority=13, startTime=1680281758916; duration=0sec 2023-03-31 16:55:59,357 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:55:59,460 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=1077 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/A/3a0829206ecb452d80ec75ace5cee6cc 2023-03-31 16:55:59,469 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:59,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 298 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35546 deadline: 1680281819463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:59,469 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:59,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 311 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35494 deadline: 1680281819464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:59,469 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:59,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 319 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35582 deadline: 1680281819465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:59,474 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=1077 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/B/57eb3818f8294a92addf3f655d000745 2023-03-31 16:55:59,478 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:59,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 300 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35496 deadline: 1680281819472, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:59,661 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:59,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 305 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35580 deadline: 1680281819657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:59,777 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:59,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 300 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35546 deadline: 1680281819770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:59,777 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:59,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 313 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35494 deadline: 1680281819771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:59,778 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:59,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 321 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35582 deadline: 1680281819772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:59,784 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:55:59,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 302 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35496 deadline: 1680281819780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:59,885 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=1077 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/C/b0d4cde27fdc4a799e013c2dcf3636a0 2023-03-31 16:55:59,889 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/A/3a0829206ecb452d80ec75ace5cee6cc as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/3a0829206ecb452d80ec75ace5cee6cc 2023-03-31 16:55:59,892 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/3a0829206ecb452d80ec75ace5cee6cc, entries=150, sequenceid=1077, filesize=11.9 K 2023-03-31 16:55:59,893 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/B/57eb3818f8294a92addf3f655d000745 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/57eb3818f8294a92addf3f655d000745 2023-03-31 16:55:59,898 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/57eb3818f8294a92addf3f655d000745, entries=150, sequenceid=1077, filesize=11.9 K 2023-03-31 16:55:59,899 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/C/b0d4cde27fdc4a799e013c2dcf3636a0 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/b0d4cde27fdc4a799e013c2dcf3636a0 2023-03-31 16:55:59,902 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/b0d4cde27fdc4a799e013c2dcf3636a0, entries=150, sequenceid=1077, filesize=11.9 K 2023-03-31 16:55:59,903 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for fd96548248d41c209e12fca5d7040af5 in 879ms, sequenceid=1077, compaction requested=false 2023-03-31 16:55:59,903 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fd96548248d41c209e12fca5d7040af5: 2023-03-31 16:55:59,903 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:55:59,903 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:55:59,903 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:55:59,904 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:59,904 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:55:59,904 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:55:59,905 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:59,905 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:59,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:59,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:59,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:59,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:59,906 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:59,906 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:55:59,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:59,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:59,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:59,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:59,907 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:55:59,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e995b5[Count = 0] remaining members to acquire global barrier 2023-03-31 16:55:59,908 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:55:59,908 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:59,912 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:59,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:59,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:59,913 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:55:59,913 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:55:59,913 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:55:59,913 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:59,913 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:55:59,919 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:59,919 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:55:59,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:59,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:59,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:59,919 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:59,919 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:55:59,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:59,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:59,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:59,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:59,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:59,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:59,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:59,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:55:59,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:55:59,922 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:55:59,922 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:55:59,922 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:55:59,922 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:55:59,922 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:55:59,923 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:59,923 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:59,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:59,924 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:59,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:59,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:59,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:55:59,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:55:59,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:59,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:55:59,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:55:59,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:59,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:59,925 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:59,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:55:59,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:59,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:59,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:55:59,928 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:59,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:55:59,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:59,931 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:55:59,931 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:55:59,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:55:59,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:55:59,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:55:59,931 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:55:59,931 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:59,931 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:59,931 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:55:59,931 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:55:59,931 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:59,931 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:55:59,931 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:55:59,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:55:59,931 DEBUG [Thread-3278] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:55:59,931 DEBUG [Thread-3278] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:55:59,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:55:59,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:00,032 DEBUG [Thread-3278] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:00,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:00,034 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:00,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:00,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:00,035 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:00,035 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:00,035 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:00,035 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:00,035 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:00,036 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:00,038 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:00,038 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:00,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:00,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:00,038 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:00,038 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:00,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:00,039 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:00,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:00,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:00,039 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:00,039 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:00,039 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:00,039 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:00,039 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:00,039 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:00,039 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:00,039 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:00,040 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:56:00,040 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. started... 2023-03-31 16:56:00,040 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fd96548248d41c209e12fca5d7040af5 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-31 16:56:00,040 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=A 2023-03-31 16:56:00,040 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:56:00,040 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=B 2023-03-31 16:56:00,040 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:56:00,040 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=C 2023-03-31 16:56:00,040 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:56:00,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on fd96548248d41c209e12fca5d7040af5 2023-03-31 16:56:00,282 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. as already flushing 2023-03-31 16:56:00,316 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:00,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 320 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35494 deadline: 1680281820309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:00,317 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680281760298, max timestamp=1680281760315Num uniques 859; 2023-03-31 16:56:00,317 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-31 16:56:00,317 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680281760298, max timestamp=1680281760315Num uniques 859; 2023-03-31 16:56:00,317 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680281760298, max timestamp=1680281760315Num uniques 859; 2023-03-31 16:56:00,317 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-31 16:56:00,317 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-31 16:56:00,360 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:00,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 332 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35582 deadline: 1680281820348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:00,368 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:00,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 311 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35496 deadline: 1680281820355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:00,369 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:00,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 312 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35546 deadline: 1680281820356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:00,423 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:00,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 322 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35494 deadline: 1680281820418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:00,450 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1089 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/A/e70ce4e78fd84f6aa637e98277c74b28 2023-03-31 16:56:00,465 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1089 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/B/157909e8edd84d69bc828fdbabd8aa05 2023-03-31 16:56:00,468 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:00,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 334 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35582 deadline: 1680281820461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:00,475 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:00,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 313 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35496 deadline: 1680281820470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:00,476 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:00,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 314 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35546 deadline: 1680281820470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:00,629 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:00,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 324 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35494 deadline: 1680281820624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:00,666 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:00,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 307 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35580 deadline: 1680281820664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:00,673 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:00,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 336 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35582 deadline: 1680281820670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:00,684 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:00,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 316 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35546 deadline: 1680281820677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:00,684 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:00,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 315 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35496 deadline: 1680281820677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:00,877 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1089 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/C/fcf6f672a32f4b568f8918d56403befe 2023-03-31 16:56:00,881 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/A/e70ce4e78fd84f6aa637e98277c74b28 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/e70ce4e78fd84f6aa637e98277c74b28 2023-03-31 16:56:00,884 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/e70ce4e78fd84f6aa637e98277c74b28, entries=150, sequenceid=1089, filesize=11.9 K 2023-03-31 16:56:00,884 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/B/157909e8edd84d69bc828fdbabd8aa05 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/157909e8edd84d69bc828fdbabd8aa05 2023-03-31 16:56:00,887 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/157909e8edd84d69bc828fdbabd8aa05, entries=150, sequenceid=1089, filesize=11.9 K 2023-03-31 16:56:00,888 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/C/fcf6f672a32f4b568f8918d56403befe as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/fcf6f672a32f4b568f8918d56403befe 2023-03-31 16:56:00,891 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/fcf6f672a32f4b568f8918d56403befe, entries=150, sequenceid=1089, filesize=11.9 K 2023-03-31 16:56:00,891 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for fd96548248d41c209e12fca5d7040af5 in 851ms, sequenceid=1089, compaction requested=true 2023-03-31 16:56:00,891 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fd96548248d41c209e12fca5d7040af5: 2023-03-31 16:56:00,891 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:56:00,892 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:00,892 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:00,892 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:00,892 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:00,892 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:00,893 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:00,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:00,893 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:00,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:00,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:00,894 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:00,894 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:00,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:00,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:00,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:00,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:00,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:00,895 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:00,895 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ba00571[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:00,895 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:00,895 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:00,896 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:00,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:00,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:00,896 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:00,896 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:00,896 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:00,896 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:00,896 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:00,898 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:00,898 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:00,898 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:00,898 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:00,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:00,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:00,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:00,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:00,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:00,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:00,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:00,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:00,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:00,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:00,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:00,900 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:00,900 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:00,900 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:00,900 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:00,900 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:00,900 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:00,903 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:00,903 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:00,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:00,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:00,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:00,903 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:00,903 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:00,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:00,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:00,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:00,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:00,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:00,904 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:00,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:00,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:00,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:00,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:00,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:00,906 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:00,906 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:00,906 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:00,906 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:00,906 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:00,906 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:00,906 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:00,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:00,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:00,906 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:00,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:00,906 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:00,907 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:00,906 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:00,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:00,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:00,907 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:00,907 DEBUG [Thread-3278] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:00,907 DEBUG [Thread-3278] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:00,907 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:56:00,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on fd96548248d41c209e12fca5d7040af5 2023-03-31 16:56:00,938 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing fd96548248d41c209e12fca5d7040af5 3/3 column families, dataSize=140.89 KB heapSize=390.84 KB 2023-03-31 16:56:00,939 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=A 2023-03-31 16:56:00,939 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:56:00,939 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=B 2023-03-31 16:56:00,939 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:56:00,939 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=C 2023-03-31 16:56:00,939 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:56:00,988 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:00,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 318 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35546 deadline: 1680281820985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:00,989 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:00,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 339 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35582 deadline: 1680281820986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:00,994 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:00,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 333 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35494 deadline: 1680281820987, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:00,994 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:00,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 317 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35496 deadline: 1680281820987, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,007 DEBUG [Thread-3278] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:01,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:01,009 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:01,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:01,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:01,010 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:01,010 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:01,010 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:01,010 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:01,011 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:01,011 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:01,013 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,013 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,013 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:01,014 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:01,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:01,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:01,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:01,019 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:01,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:01,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:01,020 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:01,023 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:01,023 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:01,023 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:01,023 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:01,023 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:01,024 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:01,024 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:01,024 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:56:01,024 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. started... 2023-03-31 16:56:01,024 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. as already flushing 2023-03-31 16:56:01,024 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:56:01,024 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:01,024 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:01,024 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,024 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:01,024 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:01,026 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,026 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:01,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:01,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:01,026 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:01,027 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:01,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:01,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:01,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:01,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:01,028 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:01,028 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a92f770[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:01,028 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:01,028 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:01,030 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:01,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:01,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:01,030 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:01,030 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:01,030 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:01,031 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,031 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:01,034 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,034 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:01,034 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:01,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:01,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:01,034 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:01,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:01,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:01,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:01,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:01,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:01,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:01,036 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:01,036 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:01,037 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:01,037 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:01,037 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:01,037 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:01,038 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:01,038 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:01,038 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:01,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:01,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:01,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:01,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:01,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:01,039 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:01,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:01,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:01,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:01,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:01,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:01,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:01,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,040 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:01,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:01,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,044 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,044 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:01,044 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:01,044 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,044 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:01,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:01,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:01,044 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:01,044 DEBUG [Thread-3278] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:01,044 DEBUG [Thread-3278] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:01,044 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:01,044 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:01,044 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:01,044 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:01,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:01,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:01,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:01,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:01,093 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:01,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 341 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35582 deadline: 1680281821090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,101 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:01,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 335 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35494 deadline: 1680281821095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,144 DEBUG [Thread-3278] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:01,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:01,146 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:01,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:01,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:01,148 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:01,148 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:01,148 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:01,148 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:01,148 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:01,148 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:01,150 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:01,150 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:01,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:01,151 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,151 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:01,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:01,151 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:01,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:01,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:01,151 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:01,152 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:01,152 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:01,152 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:01,152 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:01,152 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:01,152 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:01,152 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:01,152 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:56:01,152 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. started... 2023-03-31 16:56:01,152 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. as already flushing 2023-03-31 16:56:01,152 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:56:01,152 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:01,152 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:01,152 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,152 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:01,152 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:01,154 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,154 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:01,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:01,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:01,154 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:01,154 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:01,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:01,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:01,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:01,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:01,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:01,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6cdb8a32[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:01,156 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:01,156 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:01,157 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:01,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:01,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:01,157 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:01,157 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:01,157 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,157 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:01,157 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:01,159 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,159 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:01,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:01,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:01,159 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:01,159 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:01,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:01,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:01,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:01,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:01,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:01,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:01,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:01,161 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:01,161 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:01,161 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:01,161 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:01,162 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:01,164 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:01,164 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:01,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:01,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:01,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:01,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:01,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:01,164 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:01,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:01,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:01,165 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:01,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:01,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:01,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:01,166 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:01,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:01,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:01,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,171 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,171 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:01,171 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:01,171 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,171 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:01,171 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:01,171 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:01,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:01,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:01,171 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:01,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:01,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:01,171 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:01,171 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:01,171 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:56:01,171 DEBUG [Thread-3278] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:01,171 DEBUG [Thread-3278] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:01,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:01,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:01,271 DEBUG [Thread-3278] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:01,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:01,277 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:01,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:01,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:01,278 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:01,278 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:01,278 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:01,278 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:01,279 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:01,279 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:01,280 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:01,280 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:01,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:01,281 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,281 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:01,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:01,281 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:01,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:01,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:01,281 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:01,282 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:01,282 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:01,282 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:01,282 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:01,282 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:01,282 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:01,282 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:01,282 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:56:01,282 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. started... 2023-03-31 16:56:01,282 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. as already flushing 2023-03-31 16:56:01,282 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:56:01,283 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:01,283 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:01,283 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,283 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:01,283 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:01,284 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,284 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:01,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:01,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:01,285 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:01,285 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:01,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:01,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:01,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:01,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:01,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:01,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2053fe21[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:01,287 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:01,287 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:01,289 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:01,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:01,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:01,289 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:01,289 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:01,289 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:01,289 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,290 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:01,291 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,291 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:01,291 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:01,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:01,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:01,291 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:01,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:01,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:01,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:01,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:01,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:01,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:01,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:01,294 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:01,294 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:01,294 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:01,294 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:01,294 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:01,295 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:01,295 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:01,295 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:01,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 343 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35582 deadline: 1680281821294, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,296 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:01,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:01,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:01,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:01,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:01,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:01,296 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:01,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:01,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:01,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:01,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:01,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:01,298 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:01,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:01,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:01,300 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,300 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:01,300 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:01,300 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:01,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:01,300 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,300 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:01,300 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:01,300 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:01,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:01,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:01,300 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:01,301 DEBUG [Thread-3278] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:01,301 DEBUG [Thread-3278] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:01,300 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:01,301 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:01,301 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:56:01,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:01,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:01,305 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:01,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 337 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35494 deadline: 1680281821302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,347 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=1124 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/A/eaa4677f24544dee9ac547fa19142bde 2023-03-31 16:56:01,359 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=1124 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/B/3ae69facbdb144d2baed777fe7ca2ee9 2023-03-31 16:56:01,401 DEBUG [Thread-3278] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:01,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:01,403 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:01,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:01,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:01,404 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:01,404 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:01,404 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:01,404 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:01,405 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:01,405 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:01,410 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:01,410 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:01,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:01,410 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,410 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:01,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:01,410 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:01,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:01,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:01,411 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:01,411 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:01,411 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:01,411 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:01,411 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:01,411 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:01,411 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:01,411 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:01,412 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:56:01,412 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. started... 2023-03-31 16:56:01,412 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. as already flushing 2023-03-31 16:56:01,412 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:56:01,412 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:01,412 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:01,412 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,412 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:01,412 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:01,413 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:01,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:01,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:01,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:01,414 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:01,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:01,415 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:01,415 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:01,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:01,415 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:01,415 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b73ecd7[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:01,415 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:01,416 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:01,417 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:01,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:01,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:01,417 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:01,417 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:01,417 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:01,418 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:01,418 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,419 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:01,419 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:01,419 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:01,419 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:01,419 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:01,419 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:01,419 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:01,419 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:01,419 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:01,421 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:01,421 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:01,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:01,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:01,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:01,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:01,422 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:01,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:01,422 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:01,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:01,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:01,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:01,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:01,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:01,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:01,423 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:01,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:01,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,427 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,427 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:01,427 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:01,427 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,427 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:01,427 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:01,427 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:01,427 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:01,427 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:01,427 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:01,429 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:01,427 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:01,429 DEBUG [Thread-3278] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:01,429 DEBUG [Thread-3278] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:01,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:01,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:01,431 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:56:01,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:01,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:01,497 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:01,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 320 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35546 deadline: 1680281821493, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,497 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:01,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 319 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35496 deadline: 1680281821495, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,529 DEBUG [Thread-3278] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:01,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:01,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:01,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:01,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:01,533 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:01,533 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:01,533 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:01,533 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:01,533 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:01,533 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:01,535 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:01,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:01,535 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:01,535 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,535 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:01,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:01,535 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:01,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:01,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:01,536 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:01,536 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:01,536 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:01,536 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:01,536 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:01,536 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:01,536 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:01,536 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:01,537 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:56:01,537 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. started... 2023-03-31 16:56:01,537 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. as already flushing 2023-03-31 16:56:01,537 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:56:01,537 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:01,537 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:01,537 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,537 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:01,537 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:01,538 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:01,538 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:01,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:01,539 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:01,539 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:01,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:01,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:01,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:01,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:01,541 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:01,541 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5687b3be[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:01,541 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:01,541 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:01,543 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:01,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:01,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:01,544 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:01,544 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:01,544 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:01,544 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,544 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:01,545 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,545 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:01,545 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:01,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:01,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:01,545 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:01,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:01,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:01,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:01,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:01,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:01,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:01,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:01,549 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:01,549 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:01,549 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:01,549 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:01,549 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:01,551 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:01,551 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:01,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:01,551 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:01,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:01,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:01,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:01,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:01,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:01,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:01,552 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:01,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:01,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:01,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:01,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:01,554 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:01,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:01,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,560 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:01,560 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:01,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:01,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:01,560 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,560 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:01,560 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:01,560 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,561 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:01,561 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:01,561 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:01,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:01,561 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:01,561 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:01,561 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:01,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:01,561 DEBUG [Thread-3278] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:01,561 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:56:01,561 DEBUG [Thread-3278] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:01,603 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:01,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 345 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35582 deadline: 1680281821598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,612 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:01,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 339 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35494 deadline: 1680281821608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,661 DEBUG [Thread-3278] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:01,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:01,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:01,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:01,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:01,665 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:01,665 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:01,665 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:01,665 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:01,666 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:01,666 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:01,667 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,667 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:01,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:01,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:01,668 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,668 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:01,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:01,668 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:01,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:01,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:01,668 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:01,669 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:01,669 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:01,669 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:01,669 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:01,669 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:01,669 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:01,669 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:01,669 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:56:01,669 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. started... 2023-03-31 16:56:01,669 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. as already flushing 2023-03-31 16:56:01,669 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:56:01,669 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:01,670 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:01,670 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,670 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:01,670 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:01,671 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:01,671 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:01,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:01,672 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:01,672 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:01,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:01,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:01,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:01,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:01,675 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:01,675 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66bd3946[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:01,675 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:01,675 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:01,677 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:01,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:01,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:01,678 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:01,678 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:01,678 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:01,678 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,678 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:01,679 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,679 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:01,679 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:01,679 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:01,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:01,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:01,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:01,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:01,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:01,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:01,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:01,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:01,683 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:01,683 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:01,683 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:01,683 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:01,683 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:01,683 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:01,686 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:01,686 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:01,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:01,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:01,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:01,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:01,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:01,686 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:01,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:01,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:01,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:01,687 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:01,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:01,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:01,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:01,689 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:01,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:01,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,691 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,691 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:01,691 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:01,691 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:01,691 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,691 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:01,691 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:01,691 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:56:01,691 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:01,691 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:01,691 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:01,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:01,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:01,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:01,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:01,692 DEBUG [Thread-3278] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:01,692 DEBUG [Thread-3278] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:01,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:01,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:01,770 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=1124 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/C/1fe49092e8e541bb89a2d9be81b9963f 2023-03-31 16:56:01,774 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/A/eaa4677f24544dee9ac547fa19142bde as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/eaa4677f24544dee9ac547fa19142bde 2023-03-31 16:56:01,777 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/eaa4677f24544dee9ac547fa19142bde, entries=200, sequenceid=1124, filesize=14.3 K 2023-03-31 16:56:01,778 DEBUG [MemStoreFlusher.0] regionserver.StoreScanner(953): StoreScanner already has the close lock. There is no need to updateReaders 2023-03-31 16:56:01,778 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/B/3ae69facbdb144d2baed777fe7ca2ee9 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/3ae69facbdb144d2baed777fe7ca2ee9 2023-03-31 16:56:01,782 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/3ae69facbdb144d2baed777fe7ca2ee9, entries=150, sequenceid=1124, filesize=11.9 K 2023-03-31 16:56:01,782 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/C/1fe49092e8e541bb89a2d9be81b9963f as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/1fe49092e8e541bb89a2d9be81b9963f 2023-03-31 16:56:01,786 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/1fe49092e8e541bb89a2d9be81b9963f, entries=150, sequenceid=1124, filesize=11.9 K 2023-03-31 16:56:01,786 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~390.80 KB/400176, currentSize=46.96 KB/48090 for fd96548248d41c209e12fca5d7040af5 in 848ms, sequenceid=1124, compaction requested=true 2023-03-31 16:56:01,786 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for fd96548248d41c209e12fca5d7040af5: 2023-03-31 16:56:01,786 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:56:01,786 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 16:56:01,786 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 16:56:01,786 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:56:01,787 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:56:01,788 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 52274 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 16:56:01,788 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1912): fd96548248d41c209e12fca5d7040af5/A is initiating minor compaction (all files) 2023-03-31 16:56:01,788 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of fd96548248d41c209e12fca5d7040af5/A in TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:56:01,788 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/700a56804c0544f0a918699941cd13f4, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/3a0829206ecb452d80ec75ace5cee6cc, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/e70ce4e78fd84f6aa637e98277c74b28, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/eaa4677f24544dee9ac547fa19142bde] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp, totalSize=51.0 K 2023-03-31 16:56:01,788 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49834 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 16:56:01,788 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): fd96548248d41c209e12fca5d7040af5/B is initiating minor compaction (all files) 2023-03-31 16:56:01,788 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fd96548248d41c209e12fca5d7040af5/B in TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:56:01,788 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/2dad9ef02ddc44c1a45bc45f2f5442dd, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/57eb3818f8294a92addf3f655d000745, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/157909e8edd84d69bc828fdbabd8aa05, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/3ae69facbdb144d2baed777fe7ca2ee9] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp, totalSize=48.7 K 2023-03-31 16:56:01,789 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 2dad9ef02ddc44c1a45bc45f2f5442dd, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=1040, earliestPutTs=1680281757862 2023-03-31 16:56:01,789 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 700a56804c0544f0a918699941cd13f4, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=1040, earliestPutTs=1680281757862 2023-03-31 16:56:01,789 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 57eb3818f8294a92addf3f655d000745, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1077, earliestPutTs=1680281758500 2023-03-31 16:56:01,790 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 3a0829206ecb452d80ec75ace5cee6cc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1077, earliestPutTs=1680281758500 2023-03-31 16:56:01,790 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 157909e8edd84d69bc828fdbabd8aa05, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1089, earliestPutTs=1680281759144 2023-03-31 16:56:01,791 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting e70ce4e78fd84f6aa637e98277c74b28, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1089, earliestPutTs=1680281759144 2023-03-31 16:56:01,791 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 3ae69facbdb144d2baed777fe7ca2ee9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1124, earliestPutTs=1680281760343 2023-03-31 16:56:01,791 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting eaa4677f24544dee9ac547fa19142bde, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=1124, earliestPutTs=1680281760343 2023-03-31 16:56:01,792 DEBUG [Thread-3278] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:01,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:01,794 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:01,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:01,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:01,795 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:01,795 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:01,795 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:01,795 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:01,796 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:01,796 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:01,798 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,798 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:01,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:01,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:01,799 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:01,799 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:01,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:01,799 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:01,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:01,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:01,799 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:01,800 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:01,800 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:01,800 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:01,800 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:01,800 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:01,800 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:01,800 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:01,800 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:56:01,800 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. started... 2023-03-31 16:56:01,801 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fd96548248d41c209e12fca5d7040af5 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-31 16:56:01,801 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=A 2023-03-31 16:56:01,801 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:56:01,801 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=B 2023-03-31 16:56:01,801 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:56:01,801 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=C 2023-03-31 16:56:01,801 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:56:01,803 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): fd96548248d41c209e12fca5d7040af5#B#compaction#536 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:56:01,806 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] throttle.PressureAwareThroughputController(145): fd96548248d41c209e12fca5d7040af5#A#compaction#537 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:56:01,826 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=1134 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/A/999b80a483ac43cdb413cea783fd8bc3 2023-03-31 16:56:01,828 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/A/0221a31009fd49b3a92601782206742a as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/0221a31009fd49b3a92601782206742a 2023-03-31 16:56:01,832 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in fd96548248d41c209e12fca5d7040af5/A of fd96548248d41c209e12fca5d7040af5 into 0221a31009fd49b3a92601782206742a(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:56:01,832 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for fd96548248d41c209e12fca5d7040af5: 2023-03-31 16:56:01,832 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5., storeName=fd96548248d41c209e12fca5d7040af5/A, priority=12, startTime=1680281761786; duration=0sec 2023-03-31 16:56:01,832 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:56:01,832 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 16:56:01,833 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49834 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 16:56:01,833 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1912): fd96548248d41c209e12fca5d7040af5/C is initiating minor compaction (all files) 2023-03-31 16:56:01,833 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of fd96548248d41c209e12fca5d7040af5/C in TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:56:01,833 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/033a6a82187c4565a948c7681bdd7672, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/b0d4cde27fdc4a799e013c2dcf3636a0, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/fcf6f672a32f4b568f8918d56403befe, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/1fe49092e8e541bb89a2d9be81b9963f] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp, totalSize=48.7 K 2023-03-31 16:56:01,834 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 033a6a82187c4565a948c7681bdd7672, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=1040, earliestPutTs=1680281757862 2023-03-31 16:56:01,834 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting b0d4cde27fdc4a799e013c2dcf3636a0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1077, earliestPutTs=1680281758500 2023-03-31 16:56:01,834 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting fcf6f672a32f4b568f8918d56403befe, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1089, earliestPutTs=1680281759144 2023-03-31 16:56:01,834 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 1fe49092e8e541bb89a2d9be81b9963f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1124, earliestPutTs=1680281760343 2023-03-31 16:56:01,845 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] throttle.PressureAwareThroughputController(145): fd96548248d41c209e12fca5d7040af5#C#compaction#540 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:56:02,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on fd96548248d41c209e12fca5d7040af5 2023-03-31 16:56:02,110 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. as already flushing 2023-03-31 16:56:02,181 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680281762165, max timestamp=1680281762179Num uniques 859; 2023-03-31 16:56:02,181 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 16:56:02,181 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680281762165, max timestamp=1680281762179Num uniques 859; 2023-03-31 16:56:02,181 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 16:56:02,181 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680281762165, max timestamp=1680281762179Num uniques 859; 2023-03-31 16:56:02,181 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 16:56:02,224 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/B/c8e123f5a9d14964a4aa402b752fa9a6 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/c8e123f5a9d14964a4aa402b752fa9a6 2023-03-31 16:56:02,227 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in fd96548248d41c209e12fca5d7040af5/B of fd96548248d41c209e12fca5d7040af5 into c8e123f5a9d14964a4aa402b752fa9a6(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:56:02,227 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fd96548248d41c209e12fca5d7040af5: 2023-03-31 16:56:02,227 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5., storeName=fd96548248d41c209e12fca5d7040af5/B, priority=12, startTime=1680281761786; duration=0sec 2023-03-31 16:56:02,227 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:56:02,236 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:02,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 357 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35494 deadline: 1680281822229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,238 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:02,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 365 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35582 deadline: 1680281822230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,242 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=1134 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/B/029f7d5bb9ca4be0b28e4a86dc1543ed 2023-03-31 16:56:02,251 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=1134 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/C/f3f0b25eca744158b241b03f52084ff5 2023-03-31 16:56:02,254 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/A/999b80a483ac43cdb413cea783fd8bc3 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/999b80a483ac43cdb413cea783fd8bc3 2023-03-31 16:56:02,258 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/999b80a483ac43cdb413cea783fd8bc3, entries=150, sequenceid=1134, filesize=11.9 K 2023-03-31 16:56:02,259 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/C/5634c6545d7d40789cc308063013dd1c as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/5634c6545d7d40789cc308063013dd1c 2023-03-31 16:56:02,259 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/B/029f7d5bb9ca4be0b28e4a86dc1543ed as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/029f7d5bb9ca4be0b28e4a86dc1543ed 2023-03-31 16:56:02,263 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/029f7d5bb9ca4be0b28e4a86dc1543ed, entries=150, sequenceid=1134, filesize=11.9 K 2023-03-31 16:56:02,264 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/C/f3f0b25eca744158b241b03f52084ff5 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/f3f0b25eca744158b241b03f52084ff5 2023-03-31 16:56:02,264 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in fd96548248d41c209e12fca5d7040af5/C of fd96548248d41c209e12fca5d7040af5 into 5634c6545d7d40789cc308063013dd1c(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:56:02,264 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for fd96548248d41c209e12fca5d7040af5: 2023-03-31 16:56:02,264 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5., storeName=fd96548248d41c209e12fca5d7040af5/C, priority=12, startTime=1680281761787; duration=0sec 2023-03-31 16:56:02,264 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:56:02,267 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/f3f0b25eca744158b241b03f52084ff5, entries=150, sequenceid=1134, filesize=11.9 K 2023-03-31 16:56:02,268 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for fd96548248d41c209e12fca5d7040af5 in 467ms, sequenceid=1134, compaction requested=false 2023-03-31 16:56:02,268 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fd96548248d41c209e12fca5d7040af5: 2023-03-31 16:56:02,268 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:56:02,268 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:02,268 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:02,268 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,268 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:02,268 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:02,270 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:02,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:02,270 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:02,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:02,270 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:02,270 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:02,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:02,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:02,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:02,273 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:02,273 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59206146[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:02,273 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:02,273 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:02,274 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:02,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:02,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:02,274 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:02,274 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,274 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:02,274 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:02,274 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:02,275 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:02,276 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:02,275 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,276 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:02,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:02,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:02,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:02,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:02,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:02,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:02,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:02,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:02,279 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:02,279 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:02,279 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:02,279 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:02,279 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:02,279 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:02,281 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:02,282 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:02,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:02,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:02,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:02,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:02,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:02,282 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:02,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:02,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:02,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:02,282 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:02,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:02,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:02,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:02,283 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:02,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:02,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,287 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:02,287 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:02,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:02,287 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:02,287 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:02,287 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:02,287 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:02,287 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:02,288 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:56:02,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:02,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:02,288 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:02,288 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,288 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:02,288 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:02,288 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:02,287 DEBUG [Thread-3278] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:02,288 DEBUG [Thread-3278] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:02,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on fd96548248d41c209e12fca5d7040af5 2023-03-31 16:56:02,342 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing fd96548248d41c209e12fca5d7040af5 3/3 column families, dataSize=140.89 KB heapSize=405.61 KB 2023-03-31 16:56:02,343 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680281762222, max timestamp=1680281762340Num uniques 1018; 2023-03-31 16:56:02,343 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-03-31 16:56:02,343 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680281762222, max timestamp=1680281762340Num uniques 1018; 2023-03-31 16:56:02,343 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-03-31 16:56:02,343 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=A 2023-03-31 16:56:02,343 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-31 16:56:02,343 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=B 2023-03-31 16:56:02,343 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:56:02,343 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=C 2023-03-31 16:56:02,343 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:56:02,388 DEBUG [Thread-3278] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:02,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:02,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:02,400 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:02,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 368 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35494 deadline: 1680281822392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,400 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:02,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 376 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35582 deadline: 1680281822395, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:02,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:02,407 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:02,407 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:02,407 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:02,407 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:02,407 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:02,408 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:02,410 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,410 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:02,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:02,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:02,410 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,410 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:02,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:02,410 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:02,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:02,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:02,411 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:02,411 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:02,411 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:02,411 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:02,411 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:02,411 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:02,411 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:02,412 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:02,412 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:56:02,412 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. started... 2023-03-31 16:56:02,412 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. as already flushing 2023-03-31 16:56:02,412 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:56:02,412 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:02,412 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:02,412 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,412 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:02,412 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:02,413 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:02,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:02,413 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:02,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:02,414 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:02,414 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:02,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:02,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:02,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:02,416 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:02,416 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4da12be1[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:02,416 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:02,416 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:02,418 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:02,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:02,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:02,419 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:02,419 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:02,419 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:02,420 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,420 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:02,420 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:02,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,420 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:02,420 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,420 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:02,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:02,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:02,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:02,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:02,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:02,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:02,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:02,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:02,424 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:02,424 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:02,424 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:02,424 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:02,424 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:02,424 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:02,425 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:02,425 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:02,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:02,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:02,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:02,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:02,425 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:02,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:02,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:02,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:02,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:02,426 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:02,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:02,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:02,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:02,427 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:02,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:02,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,430 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,430 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:02,430 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:02,430 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:02,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:02,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:02,430 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,430 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:02,430 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:02,430 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:02,430 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:02,430 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:56:02,430 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:02,430 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:02,430 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:02,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:02,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:02,431 DEBUG [Thread-3278] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:02,432 DEBUG [Thread-3278] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:02,506 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:02,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 370 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35494 deadline: 1680281822501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,507 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:02,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 378 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35582 deadline: 1680281822501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,510 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:02,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 321 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35496 deadline: 1680281822503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,510 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:02,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 322 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35546 deadline: 1680281822504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,532 DEBUG [Thread-3278] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:02,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:02,534 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:02,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:02,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:02,535 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:02,535 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:02,535 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:02,535 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:02,536 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:02,536 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:02,540 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:02,540 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:02,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:02,540 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,540 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:02,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:02,541 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:02,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:02,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:02,541 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:02,541 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:02,541 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:02,541 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:02,541 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:02,542 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:02,542 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:02,542 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:02,542 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:56:02,542 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. started... 2023-03-31 16:56:02,542 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. as already flushing 2023-03-31 16:56:02,542 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:56:02,542 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:02,542 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:02,542 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,542 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:02,542 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:02,544 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,544 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:02,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:02,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:02,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:02,545 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:02,545 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:02,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:02,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:02,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:02,547 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:02,547 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26506ba8[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:02,547 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:02,547 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:02,549 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:02,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:02,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:02,549 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:02,549 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:02,550 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:02,550 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,550 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:02,552 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,552 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:02,552 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:02,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:02,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:02,552 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:02,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:02,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:02,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:02,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:02,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:02,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:02,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:02,556 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:02,556 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:02,556 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:02,556 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:02,556 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:02,558 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:02,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:02,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:02,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:02,558 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:02,558 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:02,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:02,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:02,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:02,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:02,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:02,559 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:02,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:02,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:02,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:02,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,560 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:02,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:02,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,562 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:02,562 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:02,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:02,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:02,562 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,563 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:02,563 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:02,563 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,563 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:02,563 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:02,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:02,563 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:02,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:02,563 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:02,563 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:02,563 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:02,563 DEBUG [Thread-3278] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:02,563 DEBUG [Thread-3278] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:02,663 DEBUG [Thread-3278] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:02,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:02,666 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:02,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:02,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:02,667 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:02,667 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:02,667 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:02,667 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:02,668 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:02,668 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:02,669 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,669 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:02,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:02,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:02,669 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,669 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:02,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:02,670 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:02,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:02,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:02,670 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:02,670 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:02,671 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:02,671 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:02,671 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:02,671 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:02,671 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:02,671 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:02,671 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:56:02,671 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. started... 2023-03-31 16:56:02,671 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. as already flushing 2023-03-31 16:56:02,671 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:56:02,671 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:02,671 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:02,671 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,671 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:02,671 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:02,674 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,674 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:02,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:02,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:02,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:02,674 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:02,674 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:02,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:02,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:02,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:02,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:02,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29a8112d[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:02,676 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:02,676 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:02,677 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:02,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:02,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:02,677 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:02,677 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:02,678 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:02,677 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,678 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:02,679 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,679 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:02,679 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:02,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:02,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:02,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:02,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:02,679 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:02,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:02,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:02,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:02,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,681 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:02,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:02,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 309 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35580 deadline: 1680281822676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,681 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:02,682 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:02,682 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:02,682 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:02,682 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:02,681 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:02,683 DEBUG [Thread-3270] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4172 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5., hostname=jenkins-hbase4.apache.org,33441,1680281619375, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-31 16:56:02,685 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:02,685 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:02,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:02,685 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:02,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:02,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:02,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:02,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:02,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:02,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:02,686 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:02,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:02,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:02,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:02,689 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:02,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:02,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:02,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,692 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:02,692 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,692 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:02,692 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:02,692 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:02,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:02,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:02,692 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,692 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:02,692 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:02,692 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:02,693 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:02,693 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:56:02,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:02,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:02,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:02,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:02,693 DEBUG [Thread-3278] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:02,693 DEBUG [Thread-3278] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:02,711 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:02,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 380 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35582 deadline: 1680281822708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,711 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:02,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 372 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35494 deadline: 1680281822709, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,751 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=1176 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/A/25f462d6bf864e9d948b8672cffcec8b 2023-03-31 16:56:02,793 DEBUG [Thread-3278] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:02,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:02,796 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:02,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:02,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:02,797 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:02,797 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:02,797 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:02,797 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:02,797 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:02,797 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:02,799 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:02,799 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:02,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:02,799 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,799 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:02,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:02,799 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:02,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:02,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:02,800 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:02,800 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:02,800 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:02,800 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:02,800 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:02,800 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:02,800 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:02,800 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:02,801 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:56:02,801 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. started... 2023-03-31 16:56:02,801 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. as already flushing 2023-03-31 16:56:02,801 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:56:02,801 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:02,801 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:02,801 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,801 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:02,801 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:02,802 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:02,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:02,802 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:02,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:02,803 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:02,803 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:02,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:02,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:02,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:02,804 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:02,804 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73d87f0f[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:02,804 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:02,804 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:02,806 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:02,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:02,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:02,806 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:02,806 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:02,806 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:02,806 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,806 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:02,808 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:02,808 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:02,808 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,808 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:02,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:02,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:02,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:02,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:02,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:02,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:02,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:02,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:02,811 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:02,811 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:02,811 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:02,811 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:02,811 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:02,811 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:02,813 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:02,813 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:02,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:02,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:02,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:02,813 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:02,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:02,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:02,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:02,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:02,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:02,814 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:02,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:02,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:02,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:02,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,815 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:02,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:02,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,818 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,818 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:02,818 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:02,818 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,818 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:02,818 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:02,818 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:02,818 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:02,818 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:02,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:02,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:02,818 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:02,818 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:02,818 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:02,819 DEBUG [Thread-3278] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:02,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:02,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:02,819 DEBUG [Thread-3278] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:02,919 DEBUG [Thread-3278] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:02,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:02,921 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:02,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:02,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:02,922 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:02,922 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:02,922 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:02,922 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:02,923 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:02,923 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:02,924 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,924 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:02,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:02,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:02,925 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,925 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:02,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:02,925 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:02,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:02,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:02,925 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:02,926 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:02,926 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:02,926 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:02,926 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:02,926 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:02,926 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:02,926 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:02,926 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:56:02,926 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. started... 2023-03-31 16:56:02,926 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. as already flushing 2023-03-31 16:56:02,926 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:56:02,926 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:02,926 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:02,927 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,927 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:02,927 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:02,931 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,931 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:02,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:02,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:02,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:02,931 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:02,931 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:02,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:02,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:02,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:02,932 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:02,932 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37e2bcb6[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:02,932 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:02,933 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:02,937 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:02,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:02,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:02,937 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,937 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:02,937 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:02,937 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:02,937 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:02,938 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,938 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:02,938 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:02,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:02,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:02,938 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:02,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:02,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:02,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:02,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:02,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:02,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:02,941 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:02,941 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:02,941 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:02,941 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:02,941 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:02,941 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:02,944 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:02,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:02,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:02,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:02,944 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:02,944 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:02,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:02,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:02,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:02,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:02,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:02,945 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:02,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:02,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:02,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:02,946 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:02,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:02,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,949 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,949 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:02,949 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:02,949 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:02,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:02,949 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:02,949 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:02,949 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:56:02,949 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:02,949 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:02,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:02,949 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:02,950 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:02,950 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:02,950 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:02,950 DEBUG [Thread-3278] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:02,950 DEBUG [Thread-3278] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:02,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:02,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:03,017 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:03,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 374 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35494 deadline: 1680281823012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:03,019 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:03,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 382 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35582 deadline: 1680281823014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:03,050 DEBUG [Thread-3278] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:03,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:03,052 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:03,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:03,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:03,053 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:03,054 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:03,054 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:03,054 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:03,054 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:03,054 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:03,055 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:03,055 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:03,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:03,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:03,056 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:03,056 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:03,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:03,056 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:03,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:03,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:03,057 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:03,057 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:03,057 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:03,057 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:03,057 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:03,057 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:03,057 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:03,057 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:03,057 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:56:03,057 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. started... 2023-03-31 16:56:03,059 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. as already flushing 2023-03-31 16:56:03,059 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:56:03,059 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:03,059 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:03,059 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:03,059 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:03,059 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:03,061 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:03,061 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:03,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:03,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:03,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:03,061 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:03,061 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:03,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:03,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:03,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:03,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:03,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:03,062 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:03,062 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@124a9cc9[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:03,062 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:03,063 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:03,064 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:03,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:03,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:03,064 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:03,064 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:03,064 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:03,064 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:03,064 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:03,071 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:03,071 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:03,071 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:03,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:03,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:03,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:03,071 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:03,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:03,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:03,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:03,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:03,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:03,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:03,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:03,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:03,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:03,073 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:03,074 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:03,074 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:03,074 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:03,074 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:03,075 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:03,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:03,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:03,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:03,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:03,076 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:03,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:03,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:03,076 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:03,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:03,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:03,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:03,077 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:03,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:03,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:03,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:03,077 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:03,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:03,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:03,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:03,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:03,079 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:03,079 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:03,080 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:03,080 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:03,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:03,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:03,079 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:03,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:03,080 DEBUG [Thread-3278] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:03,080 DEBUG [Thread-3278] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:03,080 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:03,080 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:03,080 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:03,080 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:03,080 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:03,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:03,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:03,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:03,161 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=1176 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/B/75e5dc56c5a344abac0c283d0ddda708 2023-03-31 16:56:03,171 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=1176 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/C/b68e1fcd4a374a63b86dea5d1d4cc0d1 2023-03-31 16:56:03,175 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/A/25f462d6bf864e9d948b8672cffcec8b as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/25f462d6bf864e9d948b8672cffcec8b 2023-03-31 16:56:03,181 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/25f462d6bf864e9d948b8672cffcec8b, entries=250, sequenceid=1176, filesize=16.7 K 2023-03-31 16:56:03,181 DEBUG [Thread-3278] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:03,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:03,183 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/B/75e5dc56c5a344abac0c283d0ddda708 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/75e5dc56c5a344abac0c283d0ddda708 2023-03-31 16:56:03,184 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:03,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:03,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:03,185 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:03,185 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:03,185 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:03,185 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:03,185 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:03,185 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:03,186 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:03,186 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:03,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:03,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:03,187 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:03,187 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:03,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:03,187 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/75e5dc56c5a344abac0c283d0ddda708, entries=150, sequenceid=1176, filesize=11.9 K 2023-03-31 16:56:03,187 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:03,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:03,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:03,188 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:03,188 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/C/b68e1fcd4a374a63b86dea5d1d4cc0d1 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/b68e1fcd4a374a63b86dea5d1d4cc0d1 2023-03-31 16:56:03,188 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:03,188 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:03,188 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:03,188 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:03,188 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:03,188 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:03,188 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:03,188 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:56:03,189 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. started... 2023-03-31 16:56:03,189 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. as already flushing 2023-03-31 16:56:03,189 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:56:03,189 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:03,189 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:03,189 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:03,189 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:03,189 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:03,191 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:03,191 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:03,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:03,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:03,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:03,192 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:03,192 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:03,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:03,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:03,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:03,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:03,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:03,194 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:03,194 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b82d6d4[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:03,194 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:03,194 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:03,194 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/b68e1fcd4a374a63b86dea5d1d4cc0d1, entries=150, sequenceid=1176, filesize=11.9 K 2023-03-31 16:56:03,195 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~71.56 KB/73280, heapSize ~198.67 KB/203440, currentSize=107.34 KB/109920 for fd96548248d41c209e12fca5d7040af5 in 852ms, sequenceid=1176, compaction requested=true 2023-03-31 16:56:03,195 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for fd96548248d41c209e12fca5d7040af5: 2023-03-31 16:56:03,195 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:56:03,195 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 16:56:03,195 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 16:56:03,195 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:56:03,195 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:03,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:03,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:03,195 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:03,195 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:03,195 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:03,195 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:03,195 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:56:03,195 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:03,196 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 42664 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 16:56:03,196 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1912): fd96548248d41c209e12fca5d7040af5/A is initiating minor compaction (all files) 2023-03-31 16:56:03,196 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of fd96548248d41c209e12fca5d7040af5/A in TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:56:03,196 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/0221a31009fd49b3a92601782206742a, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/999b80a483ac43cdb413cea783fd8bc3, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/25f462d6bf864e9d948b8672cffcec8b] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp, totalSize=41.7 K 2023-03-31 16:56:03,197 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:03,197 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:03,197 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:03,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:03,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:03,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:03,197 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:03,197 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37784 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 16:56:03,197 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 0221a31009fd49b3a92601782206742a, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=1124, earliestPutTs=1680281760343 2023-03-31 16:56:03,197 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): fd96548248d41c209e12fca5d7040af5/B is initiating minor compaction (all files) 2023-03-31 16:56:03,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:03,197 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fd96548248d41c209e12fca5d7040af5/B in TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:56:03,197 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/c8e123f5a9d14964a4aa402b752fa9a6, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/029f7d5bb9ca4be0b28e4a86dc1543ed, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/75e5dc56c5a344abac0c283d0ddda708] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp, totalSize=36.9 K 2023-03-31 16:56:03,198 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 999b80a483ac43cdb413cea783fd8bc3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1134, earliestPutTs=1680281760958 2023-03-31 16:56:03,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:03,198 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting c8e123f5a9d14964a4aa402b752fa9a6, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=1124, earliestPutTs=1680281760343 2023-03-31 16:56:03,198 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 25f462d6bf864e9d948b8672cffcec8b, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=1176, earliestPutTs=1680281762222 2023-03-31 16:56:03,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:03,198 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 029f7d5bb9ca4be0b28e4a86dc1543ed, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1134, earliestPutTs=1680281760958 2023-03-31 16:56:03,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:03,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:03,199 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 75e5dc56c5a344abac0c283d0ddda708, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1176, earliestPutTs=1680281762222 2023-03-31 16:56:03,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:03,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:03,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:03,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:03,200 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:03,200 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:03,200 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:03,200 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:03,200 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:03,203 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:03,203 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:03,203 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:03,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:03,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:03,203 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] throttle.PressureAwareThroughputController(145): fd96548248d41c209e12fca5d7040af5#A#compaction#545 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:56:03,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:03,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:03,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:03,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:03,204 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:03,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:03,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:03,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:03,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:03,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:03,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:03,206 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:03,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:03,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:03,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:03,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:03,209 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:03,209 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:03,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:03,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:03,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:03,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:03,210 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): fd96548248d41c209e12fca5d7040af5#B#compaction#546 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:56:03,211 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:03,211 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:03,211 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:03,211 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:03,211 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:03,211 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:03,211 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:03,211 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:03,211 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:03,211 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:03,211 DEBUG [Thread-3278] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:03,211 DEBUG [Thread-3278] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:03,212 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:56:03,232 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/A/1806c7fb86c8422982a60b4dacbc6b16 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/1806c7fb86c8422982a60b4dacbc6b16 2023-03-31 16:56:03,235 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/B/2d3c59483e6645718f0e7151d0bd5925 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/2d3c59483e6645718f0e7151d0bd5925 2023-03-31 16:56:03,237 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fd96548248d41c209e12fca5d7040af5/A of fd96548248d41c209e12fca5d7040af5 into 1806c7fb86c8422982a60b4dacbc6b16(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:56:03,237 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for fd96548248d41c209e12fca5d7040af5: 2023-03-31 16:56:03,237 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5., storeName=fd96548248d41c209e12fca5d7040af5/A, priority=13, startTime=1680281763195; duration=0sec 2023-03-31 16:56:03,237 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:56:03,237 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 16:56:03,238 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37784 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 16:56:03,238 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1912): fd96548248d41c209e12fca5d7040af5/C is initiating minor compaction (all files) 2023-03-31 16:56:03,238 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of fd96548248d41c209e12fca5d7040af5/C in TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:56:03,238 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/5634c6545d7d40789cc308063013dd1c, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/f3f0b25eca744158b241b03f52084ff5, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/b68e1fcd4a374a63b86dea5d1d4cc0d1] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp, totalSize=36.9 K 2023-03-31 16:56:03,239 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fd96548248d41c209e12fca5d7040af5/B of fd96548248d41c209e12fca5d7040af5 into 2d3c59483e6645718f0e7151d0bd5925(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:56:03,239 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fd96548248d41c209e12fca5d7040af5: 2023-03-31 16:56:03,239 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5., storeName=fd96548248d41c209e12fca5d7040af5/B, priority=13, startTime=1680281763195; duration=0sec 2023-03-31 16:56:03,239 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:56:03,240 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 5634c6545d7d40789cc308063013dd1c, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=1124, earliestPutTs=1680281760343 2023-03-31 16:56:03,240 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting f3f0b25eca744158b241b03f52084ff5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1134, earliestPutTs=1680281760958 2023-03-31 16:56:03,240 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting b68e1fcd4a374a63b86dea5d1d4cc0d1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1176, earliestPutTs=1680281762222 2023-03-31 16:56:03,249 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] throttle.PressureAwareThroughputController(145): fd96548248d41c209e12fca5d7040af5#C#compaction#547 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:56:03,268 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/C/22ab0458bd754741bd442a12d7fef7c5 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/22ab0458bd754741bd442a12d7fef7c5 2023-03-31 16:56:03,273 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fd96548248d41c209e12fca5d7040af5/C of fd96548248d41c209e12fca5d7040af5 into 22ab0458bd754741bd442a12d7fef7c5(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:56:03,273 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for fd96548248d41c209e12fca5d7040af5: 2023-03-31 16:56:03,273 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5., storeName=fd96548248d41c209e12fca5d7040af5/C, priority=13, startTime=1680281763195; duration=0sec 2023-03-31 16:56:03,274 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:56:03,311 DEBUG [Thread-3278] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:03,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:03,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:03,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:03,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:03,315 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:03,315 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:03,315 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:03,315 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:03,315 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:03,315 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:03,317 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:03,317 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:03,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:03,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:03,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:03,317 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:03,317 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:03,317 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:03,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:03,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:03,318 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:03,318 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:03,319 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:03,319 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:03,319 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:03,319 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:03,319 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:03,319 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:03,319 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:56:03,319 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. started... 2023-03-31 16:56:03,319 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fd96548248d41c209e12fca5d7040af5 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-31 16:56:03,319 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=A 2023-03-31 16:56:03,319 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:56:03,320 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=B 2023-03-31 16:56:03,320 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:56:03,320 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=C 2023-03-31 16:56:03,320 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:56:03,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on fd96548248d41c209e12fca5d7040af5 2023-03-31 16:56:03,523 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. as already flushing 2023-03-31 16:56:03,544 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:03,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 381 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35494 deadline: 1680281823542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:03,547 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:03,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 389 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35582 deadline: 1680281823543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:03,606 DEBUG [Thread-3285] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x10a344e6 to 127.0.0.1:51916 2023-03-31 16:56:03,606 DEBUG [Thread-3285] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 16:56:03,609 DEBUG [Thread-3283] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6ed183a4 to 127.0.0.1:51916 2023-03-31 16:56:03,609 DEBUG [Thread-3283] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 16:56:03,611 DEBUG [Thread-3287] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3a9f3972 to 127.0.0.1:51916 2023-03-31 16:56:03,611 DEBUG [Thread-3287] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 16:56:03,611 DEBUG [Thread-3281] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x549bef45 to 127.0.0.1:51916 2023-03-31 16:56:03,611 DEBUG [Thread-3281] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 16:56:03,615 DEBUG [Thread-3279] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0b475d8d to 127.0.0.1:51916 2023-03-31 16:56:03,615 DEBUG [Thread-3279] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 16:56:03,646 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:03,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 383 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35494 deadline: 1680281823646, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:03,648 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:03,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 391 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35582 deadline: 1680281823648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:03,736 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=1198 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/A/1f45c18f943d4e6cad235563fe56de2f 2023-03-31 16:56:03,746 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=1198 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/B/363a07a418d5492fa1da33f5a353d169 2023-03-31 16:56:03,847 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:03,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 385 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35494 deadline: 1680281823847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:03,849 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:03,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 393 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35582 deadline: 1680281823849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:04,151 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:04,151 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:04,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 387 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35494 deadline: 1680281824150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:04,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 395 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35582 deadline: 1680281824150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:04,154 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=1198 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/C/786c23e5d457491e87f63c7e8640c58e 2023-03-31 16:56:04,157 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/A/1f45c18f943d4e6cad235563fe56de2f as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/1f45c18f943d4e6cad235563fe56de2f 2023-03-31 16:56:04,160 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/1f45c18f943d4e6cad235563fe56de2f, entries=150, sequenceid=1198, filesize=11.9 K 2023-03-31 16:56:04,160 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/B/363a07a418d5492fa1da33f5a353d169 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/363a07a418d5492fa1da33f5a353d169 2023-03-31 16:56:04,163 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/363a07a418d5492fa1da33f5a353d169, entries=150, sequenceid=1198, filesize=11.9 K 2023-03-31 16:56:04,163 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/C/786c23e5d457491e87f63c7e8640c58e as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/786c23e5d457491e87f63c7e8640c58e 2023-03-31 16:56:04,166 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/786c23e5d457491e87f63c7e8640c58e, entries=150, sequenceid=1198, filesize=11.9 K 2023-03-31 16:56:04,166 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for fd96548248d41c209e12fca5d7040af5 in 847ms, sequenceid=1198, compaction requested=false 2023-03-31 16:56:04,167 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fd96548248d41c209e12fca5d7040af5: 2023-03-31 16:56:04,167 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:56:04,167 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:04,167 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:04,167 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:04,167 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:04,167 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:04,168 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:04,168 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:04,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:04,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:04,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:04,169 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:04,169 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:04,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:04,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:04,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:04,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:04,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:04,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:04,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b92feac[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:04,170 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:04,170 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:04,174 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:04,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:04,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:04,174 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:04,174 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:04,174 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:04,174 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:04,174 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:04,176 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:04,176 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:04,176 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:04,176 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:04,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:04,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:04,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:04,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:04,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:04,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:04,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:04,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:04,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:04,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:04,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:04,178 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:04,178 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:04,178 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:04,178 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:04,178 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:04,178 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:04,180 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:04,180 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:04,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:04,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:04,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:04,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:04,180 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:04,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:04,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:04,180 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:04,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:04,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:04,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:04,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:04,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:04,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:04,181 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:04,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:04,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:04,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:04,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:04,183 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:04,183 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:04,183 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:04,184 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:04,184 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:04,184 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:04,183 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:04,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:04,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:04,183 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:04,184 DEBUG [Thread-3278] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:04,184 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:04,184 DEBUG [Thread-3278] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:04,184 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:04,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:04,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:04,184 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:04,184 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:04,284 DEBUG [Thread-3278] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:04,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:04,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on fd96548248d41c209e12fca5d7040af5 2023-03-31 16:56:04,525 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing fd96548248d41c209e12fca5d7040af5 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-31 16:56:04,525 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=A 2023-03-31 16:56:04,525 DEBUG [Thread-3274] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x70e2be04 to 127.0.0.1:51916 2023-03-31 16:56:04,525 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:56:04,525 DEBUG [Thread-3274] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 16:56:04,525 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=B 2023-03-31 16:56:04,525 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:56:04,525 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=C 2023-03-31 16:56:04,525 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:56:04,531 DEBUG [Thread-3272] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6f2b3c03 to 127.0.0.1:51916 2023-03-31 16:56:04,531 DEBUG [Thread-3272] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 16:56:04,655 DEBUG [Thread-3268] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x50002343 to 127.0.0.1:51916 2023-03-31 16:56:04,655 DEBUG [Thread-3268] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 16:56:04,655 DEBUG [Thread-3276] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x067ea274 to 127.0.0.1:51916 2023-03-31 16:56:04,655 DEBUG [Thread-3276] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 16:56:04,931 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=1212 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/A/e49cc8193f2941189e17a43569f38320 2023-03-31 16:56:05,341 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=1212 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/B/f9ed28b19348432ca3fd5ee428b44dc8 2023-03-31 16:56:05,750 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=1212 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/C/832a2a8f745444b29665941bed9fa787 2023-03-31 16:56:05,753 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/A/e49cc8193f2941189e17a43569f38320 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/e49cc8193f2941189e17a43569f38320 2023-03-31 16:56:05,757 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/e49cc8193f2941189e17a43569f38320, entries=150, sequenceid=1212, filesize=11.9 K 2023-03-31 16:56:05,757 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/B/f9ed28b19348432ca3fd5ee428b44dc8 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/f9ed28b19348432ca3fd5ee428b44dc8 2023-03-31 16:56:05,760 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/f9ed28b19348432ca3fd5ee428b44dc8, entries=150, sequenceid=1212, filesize=11.9 K 2023-03-31 16:56:05,761 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/C/832a2a8f745444b29665941bed9fa787 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/832a2a8f745444b29665941bed9fa787 2023-03-31 16:56:05,764 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/832a2a8f745444b29665941bed9fa787, entries=150, sequenceid=1212, filesize=11.9 K 2023-03-31 16:56:05,764 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=20.13 KB/20610 for fd96548248d41c209e12fca5d7040af5 in 1239ms, sequenceid=1212, compaction requested=true 2023-03-31 16:56:05,764 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for fd96548248d41c209e12fca5d7040af5: 2023-03-31 16:56:05,764 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:56:05,765 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 16:56:05,765 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 16:56:05,765 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:56:05,765 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:56:05,765 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37886 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 16:56:05,765 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37886 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 16:56:05,765 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1912): fd96548248d41c209e12fca5d7040af5/A is initiating minor compaction (all files) 2023-03-31 16:56:05,766 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): fd96548248d41c209e12fca5d7040af5/B is initiating minor compaction (all files) 2023-03-31 16:56:05,766 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of fd96548248d41c209e12fca5d7040af5/A in TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:56:05,766 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fd96548248d41c209e12fca5d7040af5/B in TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:56:05,766 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/1806c7fb86c8422982a60b4dacbc6b16, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/1f45c18f943d4e6cad235563fe56de2f, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/e49cc8193f2941189e17a43569f38320] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp, totalSize=37.0 K 2023-03-31 16:56:05,766 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/2d3c59483e6645718f0e7151d0bd5925, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/363a07a418d5492fa1da33f5a353d169, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/f9ed28b19348432ca3fd5ee428b44dc8] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp, totalSize=37.0 K 2023-03-31 16:56:05,766 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 1806c7fb86c8422982a60b4dacbc6b16, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=1176, earliestPutTs=1680281762222 2023-03-31 16:56:05,766 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 2d3c59483e6645718f0e7151d0bd5925, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=1176, earliestPutTs=1680281762222 2023-03-31 16:56:05,766 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 1f45c18f943d4e6cad235563fe56de2f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1198, earliestPutTs=1680281762376 2023-03-31 16:56:05,766 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 363a07a418d5492fa1da33f5a353d169, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1198, earliestPutTs=1680281762376 2023-03-31 16:56:05,766 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting e49cc8193f2941189e17a43569f38320, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1212, earliestPutTs=1680281763530 2023-03-31 16:56:05,767 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting f9ed28b19348432ca3fd5ee428b44dc8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1212, earliestPutTs=1680281763530 2023-03-31 16:56:05,779 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] throttle.PressureAwareThroughputController(145): fd96548248d41c209e12fca5d7040af5#A#compaction#554 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:56:05,779 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): fd96548248d41c209e12fca5d7040af5#B#compaction#555 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:56:06,187 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/B/3f6a2533c87e40bc9c7ba49672aa0306 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/3f6a2533c87e40bc9c7ba49672aa0306 2023-03-31 16:56:06,187 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/A/efd3380443d146b19269fecbc1ebf68e as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/efd3380443d146b19269fecbc1ebf68e 2023-03-31 16:56:06,190 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fd96548248d41c209e12fca5d7040af5/A of fd96548248d41c209e12fca5d7040af5 into efd3380443d146b19269fecbc1ebf68e(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:56:06,190 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fd96548248d41c209e12fca5d7040af5/B of fd96548248d41c209e12fca5d7040af5 into 3f6a2533c87e40bc9c7ba49672aa0306(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:56:06,190 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for fd96548248d41c209e12fca5d7040af5: 2023-03-31 16:56:06,190 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fd96548248d41c209e12fca5d7040af5: 2023-03-31 16:56:06,190 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5., storeName=fd96548248d41c209e12fca5d7040af5/A, priority=13, startTime=1680281765764; duration=0sec 2023-03-31 16:56:06,190 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5., storeName=fd96548248d41c209e12fca5d7040af5/B, priority=13, startTime=1680281765765; duration=0sec 2023-03-31 16:56:06,190 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:56:06,190 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:56:06,190 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 16:56:06,191 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37886 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 16:56:06,191 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1912): fd96548248d41c209e12fca5d7040af5/C is initiating minor compaction (all files) 2023-03-31 16:56:06,191 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of fd96548248d41c209e12fca5d7040af5/C in TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:56:06,191 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/22ab0458bd754741bd442a12d7fef7c5, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/786c23e5d457491e87f63c7e8640c58e, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/832a2a8f745444b29665941bed9fa787] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp, totalSize=37.0 K 2023-03-31 16:56:06,191 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 22ab0458bd754741bd442a12d7fef7c5, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=1176, earliestPutTs=1680281762222 2023-03-31 16:56:06,192 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 786c23e5d457491e87f63c7e8640c58e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1198, earliestPutTs=1680281762376 2023-03-31 16:56:06,192 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 832a2a8f745444b29665941bed9fa787, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1212, earliestPutTs=1680281763530 2023-03-31 16:56:06,196 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] throttle.PressureAwareThroughputController(145): fd96548248d41c209e12fca5d7040af5#C#compaction#556 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:56:06,603 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/C/69a5480eeaf745009b16284dc290eb58 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/69a5480eeaf745009b16284dc290eb58 2023-03-31 16:56:06,607 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fd96548248d41c209e12fca5d7040af5/C of fd96548248d41c209e12fca5d7040af5 into 69a5480eeaf745009b16284dc290eb58(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:56:06,607 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for fd96548248d41c209e12fca5d7040af5: 2023-03-31 16:56:06,607 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5., storeName=fd96548248d41c209e12fca5d7040af5/C, priority=13, startTime=1680281765765; duration=0sec 2023-03-31 16:56:06,607 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:56:06,717 DEBUG [Thread-3270] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5726fb8b to 127.0.0.1:51916 2023-03-31 16:56:06,717 DEBUG [Thread-3270] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 16:56:06,717 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-31 16:56:06,717 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(394): wrote 244 2023-03-31 16:56:06,717 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(394): wrote 200 2023-03-31 16:56:06,717 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(394): wrote 192 2023-03-31 16:56:06,717 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(394): wrote 199 2023-03-31 16:56:06,717 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(394): wrote 254 2023-03-31 16:56:06,717 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-31 16:56:06,717 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-31 16:56:06,717 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(402): scanned 1635 2023-03-31 16:56:06,718 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(403): verified 4883 rows 2023-03-31 16:56:06,718 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(402): scanned 1633 2023-03-31 16:56:06,718 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(403): verified 4898 rows 2023-03-31 16:56:06,718 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(402): scanned 1630 2023-03-31 16:56:06,718 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(403): verified 4885 rows 2023-03-31 16:56:06,718 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(402): scanned 1623 2023-03-31 16:56:06,718 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(403): verified 4868 rows 2023-03-31 16:56:06,718 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(402): scanned 1638 2023-03-31 16:56:06,718 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(403): verified 4906 rows 2023-03-31 16:56:06,718 INFO [Listener at localhost/44135] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-31 16:56:06,718 DEBUG [Listener at localhost/44135] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x20ed9117 to 127.0.0.1:51916 2023-03-31 16:56:06,718 DEBUG [Listener at localhost/44135] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 16:56:06,722 INFO [Listener at localhost/44135] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-31 16:56:06,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-03-31 16:56:06,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure2.ProcedureExecutor(1029): Stored pid=50, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-31 16:56:06,727 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680281766727"}]},"ts":"1680281766727"} 2023-03-31 16:56:06,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-31 16:56:06,728 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-31 16:56:06,826 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-31 16:56:06,827 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=fd96548248d41c209e12fca5d7040af5, UNASSIGN}] 2023-03-31 16:56:06,828 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=fd96548248d41c209e12fca5d7040af5, UNASSIGN 2023-03-31 16:56:06,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-31 16:56:06,829 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=fd96548248d41c209e12fca5d7040af5, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:06,829 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680281766829"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680281766829"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680281766829"}]},"ts":"1680281766829"} 2023-03-31 16:56:06,830 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=52, ppid=51, state=RUNNABLE; CloseRegionProcedure fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375}] 2023-03-31 16:56:06,981 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close fd96548248d41c209e12fca5d7040af5 2023-03-31 16:56:06,981 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing fd96548248d41c209e12fca5d7040af5, disabling compactions & flushes 2023-03-31 16:56:06,981 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:56:06,981 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:56:06,981 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. after waiting 0 ms 2023-03-31 16:56:06,981 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:56:06,982 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing fd96548248d41c209e12fca5d7040af5 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-03-31 16:56:06,982 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=A 2023-03-31 16:56:06,982 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:56:06,982 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=B 2023-03-31 16:56:06,982 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:56:06,982 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fd96548248d41c209e12fca5d7040af5, store=C 2023-03-31 16:56:06,982 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:56:07,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-31 16:56:07,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-31 16:56:07,388 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1222 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/A/32c17e08f748497ab858c33ef63ca971 2023-03-31 16:56:07,796 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1222 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/B/7e4a15b332764af191b84c28a2898da8 2023-03-31 16:56:07,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-31 16:56:08,204 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1222 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/C/457610069ccc423eaa93c90cebe213df 2023-03-31 16:56:08,207 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/A/32c17e08f748497ab858c33ef63ca971 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/32c17e08f748497ab858c33ef63ca971 2023-03-31 16:56:08,210 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/32c17e08f748497ab858c33ef63ca971, entries=150, sequenceid=1222, filesize=11.9 K 2023-03-31 16:56:08,210 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/B/7e4a15b332764af191b84c28a2898da8 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/7e4a15b332764af191b84c28a2898da8 2023-03-31 16:56:08,213 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/7e4a15b332764af191b84c28a2898da8, entries=150, sequenceid=1222, filesize=11.9 K 2023-03-31 16:56:08,214 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/.tmp/C/457610069ccc423eaa93c90cebe213df as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/457610069ccc423eaa93c90cebe213df 2023-03-31 16:56:08,217 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/457610069ccc423eaa93c90cebe213df, entries=150, sequenceid=1222, filesize=11.9 K 2023-03-31 16:56:08,217 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for fd96548248d41c209e12fca5d7040af5 in 1236ms, sequenceid=1222, compaction requested=false 2023-03-31 16:56:08,240 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/daf61565fc3f4daca4e22f65cc78999d, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/51096b5d8f38425e965ecdb0dedda48a, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/d184f08a38ab4d9c87a2587b0625f704, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/5a1b5b8cb60e476d9fe4141b68065b00, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/94e1a482c06f44fd8377ad9b367d1a7c, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/ba96916252d646d6bdfaf90176c04f05, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/4161aa1256cf47f6badfcb397959f7e4, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/8c5edf7daad04a9e96b32cc8a516d5e3, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/ac1a7331a0ee480abd994c5979aa85af, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/1af39476ae1e488794a4a84669d8b770, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/113379a1323548739914b0e988e3ab46, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/333a6544de9041a9a9272fbdb4a22627, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/733dbb73607a4da588aaff5ce014c34c, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/13818983462f4ee9a100843f5c7e7a63, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/a311336456b9497a86565ffef7c50086, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/be75b33618904778a2f9484b9332aade, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/408b46ee2c4f41ec9aea29a7e9bb7e90, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/8a49671736d149209dbb5b3b9dba8ada, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/36742c66d019482b8c8307f42de79c3c, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/196c3e427396455aaa23cd195ebd0802, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/2617ae3f4d7a44d585c8512e9eaae28e, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/1ca8bc1c03fc4bb8ab992fe817a203c6, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/834a0c767616459084dfac42dd3ad093, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/9b62fccf96fc4ea2bb4f07713b154338, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/270b468650f94952a62615d1c8e93190, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/8badd180752f4adeb892fe6382b5ebbc, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/268709b3753c4c19b2398237fec79d6e, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/f5c4ef4e600047b9acd2570278f100f4, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/d95c5e11761941ad996f1190345a0f82, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/cf6deddab5a049de9c38d7e4fbedab28, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/014712c3f1bc49e3bf4e68b0c826ad38, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/8fd2c9059f59463eac1393cbebd2509f, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/700a56804c0544f0a918699941cd13f4, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/3a0829206ecb452d80ec75ace5cee6cc, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/e70ce4e78fd84f6aa637e98277c74b28, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/eaa4677f24544dee9ac547fa19142bde, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/0221a31009fd49b3a92601782206742a, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/999b80a483ac43cdb413cea783fd8bc3, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/25f462d6bf864e9d948b8672cffcec8b, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/1806c7fb86c8422982a60b4dacbc6b16, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/1f45c18f943d4e6cad235563fe56de2f, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/e49cc8193f2941189e17a43569f38320] to archive 2023-03-31 16:56:08,241 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-31 16:56:08,243 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/daf61565fc3f4daca4e22f65cc78999d to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/daf61565fc3f4daca4e22f65cc78999d 2023-03-31 16:56:08,243 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/51096b5d8f38425e965ecdb0dedda48a to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/51096b5d8f38425e965ecdb0dedda48a 2023-03-31 16:56:08,244 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/d184f08a38ab4d9c87a2587b0625f704 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/d184f08a38ab4d9c87a2587b0625f704 2023-03-31 16:56:08,245 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/5a1b5b8cb60e476d9fe4141b68065b00 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/5a1b5b8cb60e476d9fe4141b68065b00 2023-03-31 16:56:08,246 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/94e1a482c06f44fd8377ad9b367d1a7c to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/94e1a482c06f44fd8377ad9b367d1a7c 2023-03-31 16:56:08,247 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/ba96916252d646d6bdfaf90176c04f05 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/ba96916252d646d6bdfaf90176c04f05 2023-03-31 16:56:08,248 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/4161aa1256cf47f6badfcb397959f7e4 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/4161aa1256cf47f6badfcb397959f7e4 2023-03-31 16:56:08,248 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/8c5edf7daad04a9e96b32cc8a516d5e3 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/8c5edf7daad04a9e96b32cc8a516d5e3 2023-03-31 16:56:08,249 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/ac1a7331a0ee480abd994c5979aa85af to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/ac1a7331a0ee480abd994c5979aa85af 2023-03-31 16:56:08,250 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/1af39476ae1e488794a4a84669d8b770 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/1af39476ae1e488794a4a84669d8b770 2023-03-31 16:56:08,251 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/113379a1323548739914b0e988e3ab46 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/113379a1323548739914b0e988e3ab46 2023-03-31 16:56:08,252 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/333a6544de9041a9a9272fbdb4a22627 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/333a6544de9041a9a9272fbdb4a22627 2023-03-31 16:56:08,253 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/733dbb73607a4da588aaff5ce014c34c to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/733dbb73607a4da588aaff5ce014c34c 2023-03-31 16:56:08,254 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/13818983462f4ee9a100843f5c7e7a63 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/13818983462f4ee9a100843f5c7e7a63 2023-03-31 16:56:08,254 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/a311336456b9497a86565ffef7c50086 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/a311336456b9497a86565ffef7c50086 2023-03-31 16:56:08,255 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/be75b33618904778a2f9484b9332aade to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/be75b33618904778a2f9484b9332aade 2023-03-31 16:56:08,256 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/408b46ee2c4f41ec9aea29a7e9bb7e90 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/408b46ee2c4f41ec9aea29a7e9bb7e90 2023-03-31 16:56:08,257 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/8a49671736d149209dbb5b3b9dba8ada to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/8a49671736d149209dbb5b3b9dba8ada 2023-03-31 16:56:08,258 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/36742c66d019482b8c8307f42de79c3c to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/36742c66d019482b8c8307f42de79c3c 2023-03-31 16:56:08,259 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/196c3e427396455aaa23cd195ebd0802 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/196c3e427396455aaa23cd195ebd0802 2023-03-31 16:56:08,259 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/2617ae3f4d7a44d585c8512e9eaae28e to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/2617ae3f4d7a44d585c8512e9eaae28e 2023-03-31 16:56:08,260 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/1ca8bc1c03fc4bb8ab992fe817a203c6 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/1ca8bc1c03fc4bb8ab992fe817a203c6 2023-03-31 16:56:08,261 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/834a0c767616459084dfac42dd3ad093 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/834a0c767616459084dfac42dd3ad093 2023-03-31 16:56:08,262 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/9b62fccf96fc4ea2bb4f07713b154338 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/9b62fccf96fc4ea2bb4f07713b154338 2023-03-31 16:56:08,263 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/270b468650f94952a62615d1c8e93190 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/270b468650f94952a62615d1c8e93190 2023-03-31 16:56:08,264 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/8badd180752f4adeb892fe6382b5ebbc to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/8badd180752f4adeb892fe6382b5ebbc 2023-03-31 16:56:08,265 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/268709b3753c4c19b2398237fec79d6e to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/268709b3753c4c19b2398237fec79d6e 2023-03-31 16:56:08,266 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/f5c4ef4e600047b9acd2570278f100f4 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/f5c4ef4e600047b9acd2570278f100f4 2023-03-31 16:56:08,266 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/d95c5e11761941ad996f1190345a0f82 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/d95c5e11761941ad996f1190345a0f82 2023-03-31 16:56:08,267 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/cf6deddab5a049de9c38d7e4fbedab28 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/cf6deddab5a049de9c38d7e4fbedab28 2023-03-31 16:56:08,268 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/014712c3f1bc49e3bf4e68b0c826ad38 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/014712c3f1bc49e3bf4e68b0c826ad38 2023-03-31 16:56:08,269 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/8fd2c9059f59463eac1393cbebd2509f to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/8fd2c9059f59463eac1393cbebd2509f 2023-03-31 16:56:08,270 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/700a56804c0544f0a918699941cd13f4 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/700a56804c0544f0a918699941cd13f4 2023-03-31 16:56:08,271 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/3a0829206ecb452d80ec75ace5cee6cc to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/3a0829206ecb452d80ec75ace5cee6cc 2023-03-31 16:56:08,272 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/e70ce4e78fd84f6aa637e98277c74b28 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/e70ce4e78fd84f6aa637e98277c74b28 2023-03-31 16:56:08,273 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/eaa4677f24544dee9ac547fa19142bde to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/eaa4677f24544dee9ac547fa19142bde 2023-03-31 16:56:08,274 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/0221a31009fd49b3a92601782206742a to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/0221a31009fd49b3a92601782206742a 2023-03-31 16:56:08,275 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/999b80a483ac43cdb413cea783fd8bc3 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/999b80a483ac43cdb413cea783fd8bc3 2023-03-31 16:56:08,276 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/25f462d6bf864e9d948b8672cffcec8b to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/25f462d6bf864e9d948b8672cffcec8b 2023-03-31 16:56:08,277 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/1806c7fb86c8422982a60b4dacbc6b16 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/1806c7fb86c8422982a60b4dacbc6b16 2023-03-31 16:56:08,278 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/1f45c18f943d4e6cad235563fe56de2f to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/1f45c18f943d4e6cad235563fe56de2f 2023-03-31 16:56:08,279 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/e49cc8193f2941189e17a43569f38320 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/e49cc8193f2941189e17a43569f38320 2023-03-31 16:56:08,282 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/d7a06737ddc9484f8e960bb281027c0d, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/71de6e7c03bf4dd386c62fe1369efaf8, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/a8c313e2aa4e48f693d9d3c03dc53c27, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/c300e3d5f7404822a98b0ed4da06b338, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/aaeab0834e474f02bd12099b13627d9a, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/c6883da5c49848d1879d316b019fd300, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/dcee5968ec6d4db98315146693565e29, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/353ad655953f4c29bb1028c2855023ff, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/4f80bbf865894e77b40725b14fda87a8, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/07ac830d72144b50ac986ad6ca9c61ee, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/45bd50b8c4534d3eac4b65d71bfb3d38, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/2bdb0f8176e94f3880f8e6441adcc336, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/9146e335fac246dc94fdd2a6c259335c, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/a323f04d9f0d484cbe71c78cbecdb1e4, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/c6d85b68e0024b34bc43362dd074eb22, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/a9ca0c73aaec4967b43bfb0fc7ca016e, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/ec9c1ce3409245e6b22c33edac13ea86, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/8f878fabbb954e4fa775f51764f84cf7, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/b7d99f8d1066467b8dea50a57d8b87c7, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/228bc33de14f481390094dee19dbf242, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/cfb5ffd9bdbb42018e249743caa25b00, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/0cf3ffffde0541c6bb2eb23a85eae8ab, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/03352426d4e543c2a58b4f37f24fa868, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/f667690121dc48cf8269b6482f058333, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/108601ebd8e44dcd801810eb2057018c, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/a6327bc1494345408e341be5c6cefe9f, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/ae10b461a8634c0db33bef333b6fdec8, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/ac86e89d9098437d89899f7e27cf7d7e, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/68d5852e59c7436281da1157e0363497, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/4a56398418664a7e93addb403d9e0a5f, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/cea9a4e7da7847f0af7d82800c562d22, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/2dad9ef02ddc44c1a45bc45f2f5442dd, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/2e378fdcfd9846b6af7146900eccaede, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/57eb3818f8294a92addf3f655d000745, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/157909e8edd84d69bc828fdbabd8aa05, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/c8e123f5a9d14964a4aa402b752fa9a6, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/3ae69facbdb144d2baed777fe7ca2ee9, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/029f7d5bb9ca4be0b28e4a86dc1543ed, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/2d3c59483e6645718f0e7151d0bd5925, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/75e5dc56c5a344abac0c283d0ddda708, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/363a07a418d5492fa1da33f5a353d169, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/f9ed28b19348432ca3fd5ee428b44dc8] to archive 2023-03-31 16:56:08,282 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-31 16:56:08,284 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/d7a06737ddc9484f8e960bb281027c0d to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/d7a06737ddc9484f8e960bb281027c0d 2023-03-31 16:56:08,285 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/71de6e7c03bf4dd386c62fe1369efaf8 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/71de6e7c03bf4dd386c62fe1369efaf8 2023-03-31 16:56:08,285 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/a8c313e2aa4e48f693d9d3c03dc53c27 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/a8c313e2aa4e48f693d9d3c03dc53c27 2023-03-31 16:56:08,286 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/c300e3d5f7404822a98b0ed4da06b338 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/c300e3d5f7404822a98b0ed4da06b338 2023-03-31 16:56:08,287 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/aaeab0834e474f02bd12099b13627d9a to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/aaeab0834e474f02bd12099b13627d9a 2023-03-31 16:56:08,288 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/c6883da5c49848d1879d316b019fd300 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/c6883da5c49848d1879d316b019fd300 2023-03-31 16:56:08,289 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/dcee5968ec6d4db98315146693565e29 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/dcee5968ec6d4db98315146693565e29 2023-03-31 16:56:08,290 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/353ad655953f4c29bb1028c2855023ff to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/353ad655953f4c29bb1028c2855023ff 2023-03-31 16:56:08,291 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/4f80bbf865894e77b40725b14fda87a8 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/4f80bbf865894e77b40725b14fda87a8 2023-03-31 16:56:08,291 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/07ac830d72144b50ac986ad6ca9c61ee to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/07ac830d72144b50ac986ad6ca9c61ee 2023-03-31 16:56:08,292 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/45bd50b8c4534d3eac4b65d71bfb3d38 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/45bd50b8c4534d3eac4b65d71bfb3d38 2023-03-31 16:56:08,293 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/2bdb0f8176e94f3880f8e6441adcc336 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/2bdb0f8176e94f3880f8e6441adcc336 2023-03-31 16:56:08,294 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/9146e335fac246dc94fdd2a6c259335c to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/9146e335fac246dc94fdd2a6c259335c 2023-03-31 16:56:08,295 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/a323f04d9f0d484cbe71c78cbecdb1e4 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/a323f04d9f0d484cbe71c78cbecdb1e4 2023-03-31 16:56:08,296 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/c6d85b68e0024b34bc43362dd074eb22 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/c6d85b68e0024b34bc43362dd074eb22 2023-03-31 16:56:08,296 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/a9ca0c73aaec4967b43bfb0fc7ca016e to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/a9ca0c73aaec4967b43bfb0fc7ca016e 2023-03-31 16:56:08,297 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/ec9c1ce3409245e6b22c33edac13ea86 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/ec9c1ce3409245e6b22c33edac13ea86 2023-03-31 16:56:08,298 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/8f878fabbb954e4fa775f51764f84cf7 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/8f878fabbb954e4fa775f51764f84cf7 2023-03-31 16:56:08,299 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/b7d99f8d1066467b8dea50a57d8b87c7 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/b7d99f8d1066467b8dea50a57d8b87c7 2023-03-31 16:56:08,299 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/228bc33de14f481390094dee19dbf242 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/228bc33de14f481390094dee19dbf242 2023-03-31 16:56:08,300 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/cfb5ffd9bdbb42018e249743caa25b00 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/cfb5ffd9bdbb42018e249743caa25b00 2023-03-31 16:56:08,301 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/0cf3ffffde0541c6bb2eb23a85eae8ab to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/0cf3ffffde0541c6bb2eb23a85eae8ab 2023-03-31 16:56:08,302 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/03352426d4e543c2a58b4f37f24fa868 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/03352426d4e543c2a58b4f37f24fa868 2023-03-31 16:56:08,302 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/f667690121dc48cf8269b6482f058333 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/f667690121dc48cf8269b6482f058333 2023-03-31 16:56:08,303 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/108601ebd8e44dcd801810eb2057018c to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/108601ebd8e44dcd801810eb2057018c 2023-03-31 16:56:08,304 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/a6327bc1494345408e341be5c6cefe9f to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/a6327bc1494345408e341be5c6cefe9f 2023-03-31 16:56:08,305 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/ae10b461a8634c0db33bef333b6fdec8 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/ae10b461a8634c0db33bef333b6fdec8 2023-03-31 16:56:08,305 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/ac86e89d9098437d89899f7e27cf7d7e to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/ac86e89d9098437d89899f7e27cf7d7e 2023-03-31 16:56:08,306 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/68d5852e59c7436281da1157e0363497 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/68d5852e59c7436281da1157e0363497 2023-03-31 16:56:08,307 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/4a56398418664a7e93addb403d9e0a5f to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/4a56398418664a7e93addb403d9e0a5f 2023-03-31 16:56:08,308 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/cea9a4e7da7847f0af7d82800c562d22 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/cea9a4e7da7847f0af7d82800c562d22 2023-03-31 16:56:08,308 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/2dad9ef02ddc44c1a45bc45f2f5442dd to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/2dad9ef02ddc44c1a45bc45f2f5442dd 2023-03-31 16:56:08,309 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/2e378fdcfd9846b6af7146900eccaede to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/2e378fdcfd9846b6af7146900eccaede 2023-03-31 16:56:08,310 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/57eb3818f8294a92addf3f655d000745 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/57eb3818f8294a92addf3f655d000745 2023-03-31 16:56:08,311 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/157909e8edd84d69bc828fdbabd8aa05 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/157909e8edd84d69bc828fdbabd8aa05 2023-03-31 16:56:08,312 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/c8e123f5a9d14964a4aa402b752fa9a6 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/c8e123f5a9d14964a4aa402b752fa9a6 2023-03-31 16:56:08,313 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/3ae69facbdb144d2baed777fe7ca2ee9 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/3ae69facbdb144d2baed777fe7ca2ee9 2023-03-31 16:56:08,314 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/029f7d5bb9ca4be0b28e4a86dc1543ed to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/029f7d5bb9ca4be0b28e4a86dc1543ed 2023-03-31 16:56:08,315 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/2d3c59483e6645718f0e7151d0bd5925 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/2d3c59483e6645718f0e7151d0bd5925 2023-03-31 16:56:08,316 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/75e5dc56c5a344abac0c283d0ddda708 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/75e5dc56c5a344abac0c283d0ddda708 2023-03-31 16:56:08,316 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/363a07a418d5492fa1da33f5a353d169 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/363a07a418d5492fa1da33f5a353d169 2023-03-31 16:56:08,317 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/f9ed28b19348432ca3fd5ee428b44dc8 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/f9ed28b19348432ca3fd5ee428b44dc8 2023-03-31 16:56:08,320 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/ccb45276e3cc44bdb5f0d7c25af6246f, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/d7cd4d3f7efb4531b652c72ef9a96f20, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/7e094804cabc48b69e71f77925e5c460, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/f1a52c170c6942ad8fcd34df8610daff, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/5cb23aca555645fba28db8a653d0813b, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/9899463f779d4f01b9faaf75b7339b58, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/91677ed0537c4217b140719c20221425, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/9e45e98286c94a5dbd25d44026aa9e03, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/ae5fb4495bd0447d97f566cdc588ae6b, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/7b2b2909a0a34c9097176986b337ba82, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/ec7afcab177f478c958452d0aff83946, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/5ed60d32bc9c40a59ce00c804b0f3f90, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/90253ba9fc1b4b069c9741467cdaf888, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/0d1d9f722a6542ee9feaa02e210b8abc, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/de63658e2b154329b801542b6bd4d830, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/b2bf5ac1a7484e68ae4427104f0644a6, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/5d31905e36334081acef1152f9f95857, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/7f704258fa4b49f893af6fb804382787, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/fe0e4e08756b4b7092da9936163f236a, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/64a381552e44443dacb98d2591a749a5, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/ac043966fc7448c0972e50fb65d82bf7, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/c290e38f115345dc989b8300f91780b6, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/5de85fbfe91646239c8619e33f83e550, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/17044945c73e48ec8927e37589b03625, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/fca5c0f726ad48d880420c188dce0c56, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/e5e438dc0e6f4a3bb60c5a64237e7c46, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/0d1daaf9d7e54697abb38789b1fd1350, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/774707c3dcbd4d83a8a6c044b07efe1d, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/83f492e3c71c4a7cb0e07753a923555f, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/2864308f27f545a09962fb08fe7eca01, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/dd6edc3e7b294e45a993e1b8ae74bf6a, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/033a6a82187c4565a948c7681bdd7672, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/c87d16ea699f4075ab53b0a18f2d9821, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/b0d4cde27fdc4a799e013c2dcf3636a0, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/fcf6f672a32f4b568f8918d56403befe, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/5634c6545d7d40789cc308063013dd1c, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/1fe49092e8e541bb89a2d9be81b9963f, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/f3f0b25eca744158b241b03f52084ff5, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/22ab0458bd754741bd442a12d7fef7c5, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/b68e1fcd4a374a63b86dea5d1d4cc0d1, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/786c23e5d457491e87f63c7e8640c58e, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/832a2a8f745444b29665941bed9fa787] to archive 2023-03-31 16:56:08,321 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-31 16:56:08,322 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/ccb45276e3cc44bdb5f0d7c25af6246f to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/ccb45276e3cc44bdb5f0d7c25af6246f 2023-03-31 16:56:08,323 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/d7cd4d3f7efb4531b652c72ef9a96f20 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/d7cd4d3f7efb4531b652c72ef9a96f20 2023-03-31 16:56:08,324 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/7e094804cabc48b69e71f77925e5c460 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/7e094804cabc48b69e71f77925e5c460 2023-03-31 16:56:08,325 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/f1a52c170c6942ad8fcd34df8610daff to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/f1a52c170c6942ad8fcd34df8610daff 2023-03-31 16:56:08,326 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/5cb23aca555645fba28db8a653d0813b to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/5cb23aca555645fba28db8a653d0813b 2023-03-31 16:56:08,327 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/9899463f779d4f01b9faaf75b7339b58 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/9899463f779d4f01b9faaf75b7339b58 2023-03-31 16:56:08,327 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/91677ed0537c4217b140719c20221425 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/91677ed0537c4217b140719c20221425 2023-03-31 16:56:08,328 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/9e45e98286c94a5dbd25d44026aa9e03 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/9e45e98286c94a5dbd25d44026aa9e03 2023-03-31 16:56:08,329 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/ae5fb4495bd0447d97f566cdc588ae6b to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/ae5fb4495bd0447d97f566cdc588ae6b 2023-03-31 16:56:08,330 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/7b2b2909a0a34c9097176986b337ba82 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/7b2b2909a0a34c9097176986b337ba82 2023-03-31 16:56:08,331 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/ec7afcab177f478c958452d0aff83946 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/ec7afcab177f478c958452d0aff83946 2023-03-31 16:56:08,332 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/5ed60d32bc9c40a59ce00c804b0f3f90 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/5ed60d32bc9c40a59ce00c804b0f3f90 2023-03-31 16:56:08,333 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/90253ba9fc1b4b069c9741467cdaf888 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/90253ba9fc1b4b069c9741467cdaf888 2023-03-31 16:56:08,333 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/0d1d9f722a6542ee9feaa02e210b8abc to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/0d1d9f722a6542ee9feaa02e210b8abc 2023-03-31 16:56:08,334 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/de63658e2b154329b801542b6bd4d830 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/de63658e2b154329b801542b6bd4d830 2023-03-31 16:56:08,335 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/b2bf5ac1a7484e68ae4427104f0644a6 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/b2bf5ac1a7484e68ae4427104f0644a6 2023-03-31 16:56:08,336 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/5d31905e36334081acef1152f9f95857 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/5d31905e36334081acef1152f9f95857 2023-03-31 16:56:08,337 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/7f704258fa4b49f893af6fb804382787 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/7f704258fa4b49f893af6fb804382787 2023-03-31 16:56:08,338 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/fe0e4e08756b4b7092da9936163f236a to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/fe0e4e08756b4b7092da9936163f236a 2023-03-31 16:56:08,339 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/64a381552e44443dacb98d2591a749a5 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/64a381552e44443dacb98d2591a749a5 2023-03-31 16:56:08,339 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/ac043966fc7448c0972e50fb65d82bf7 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/ac043966fc7448c0972e50fb65d82bf7 2023-03-31 16:56:08,340 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/c290e38f115345dc989b8300f91780b6 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/c290e38f115345dc989b8300f91780b6 2023-03-31 16:56:08,341 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/5de85fbfe91646239c8619e33f83e550 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/5de85fbfe91646239c8619e33f83e550 2023-03-31 16:56:08,342 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/17044945c73e48ec8927e37589b03625 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/17044945c73e48ec8927e37589b03625 2023-03-31 16:56:08,343 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/fca5c0f726ad48d880420c188dce0c56 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/fca5c0f726ad48d880420c188dce0c56 2023-03-31 16:56:08,344 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/e5e438dc0e6f4a3bb60c5a64237e7c46 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/e5e438dc0e6f4a3bb60c5a64237e7c46 2023-03-31 16:56:08,345 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/0d1daaf9d7e54697abb38789b1fd1350 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/0d1daaf9d7e54697abb38789b1fd1350 2023-03-31 16:56:08,345 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/774707c3dcbd4d83a8a6c044b07efe1d to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/774707c3dcbd4d83a8a6c044b07efe1d 2023-03-31 16:56:08,346 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/83f492e3c71c4a7cb0e07753a923555f to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/83f492e3c71c4a7cb0e07753a923555f 2023-03-31 16:56:08,347 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/2864308f27f545a09962fb08fe7eca01 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/2864308f27f545a09962fb08fe7eca01 2023-03-31 16:56:08,348 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/dd6edc3e7b294e45a993e1b8ae74bf6a to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/dd6edc3e7b294e45a993e1b8ae74bf6a 2023-03-31 16:56:08,349 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/033a6a82187c4565a948c7681bdd7672 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/033a6a82187c4565a948c7681bdd7672 2023-03-31 16:56:08,350 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/c87d16ea699f4075ab53b0a18f2d9821 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/c87d16ea699f4075ab53b0a18f2d9821 2023-03-31 16:56:08,350 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/b0d4cde27fdc4a799e013c2dcf3636a0 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/b0d4cde27fdc4a799e013c2dcf3636a0 2023-03-31 16:56:08,351 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/fcf6f672a32f4b568f8918d56403befe to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/fcf6f672a32f4b568f8918d56403befe 2023-03-31 16:56:08,352 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/5634c6545d7d40789cc308063013dd1c to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/5634c6545d7d40789cc308063013dd1c 2023-03-31 16:56:08,353 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/1fe49092e8e541bb89a2d9be81b9963f to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/1fe49092e8e541bb89a2d9be81b9963f 2023-03-31 16:56:08,354 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/f3f0b25eca744158b241b03f52084ff5 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/f3f0b25eca744158b241b03f52084ff5 2023-03-31 16:56:08,355 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/22ab0458bd754741bd442a12d7fef7c5 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/22ab0458bd754741bd442a12d7fef7c5 2023-03-31 16:56:08,355 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/b68e1fcd4a374a63b86dea5d1d4cc0d1 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/b68e1fcd4a374a63b86dea5d1d4cc0d1 2023-03-31 16:56:08,356 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/786c23e5d457491e87f63c7e8640c58e to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/786c23e5d457491e87f63c7e8640c58e 2023-03-31 16:56:08,357 DEBUG [StoreCloser-TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/832a2a8f745444b29665941bed9fa787 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/832a2a8f745444b29665941bed9fa787 2023-03-31 16:56:08,361 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/recovered.edits/1225.seqid, newMaxSeqId=1225, maxSeqId=1 2023-03-31 16:56:08,362 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5. 2023-03-31 16:56:08,362 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for fd96548248d41c209e12fca5d7040af5: 2023-03-31 16:56:08,364 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed fd96548248d41c209e12fca5d7040af5 2023-03-31 16:56:08,364 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=fd96548248d41c209e12fca5d7040af5, regionState=CLOSED 2023-03-31 16:56:08,364 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680281768364"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680281768364"}]},"ts":"1680281768364"} 2023-03-31 16:56:08,366 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=52, resume processing ppid=51 2023-03-31 16:56:08,366 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=52, ppid=51, state=SUCCESS; CloseRegionProcedure fd96548248d41c209e12fca5d7040af5, server=jenkins-hbase4.apache.org,33441,1680281619375 in 1.5350 sec 2023-03-31 16:56:08,367 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=51, resume processing ppid=50 2023-03-31 16:56:08,367 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=50, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=fd96548248d41c209e12fca5d7040af5, UNASSIGN in 1.5390 sec 2023-03-31 16:56:08,367 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680281768367"}]},"ts":"1680281768367"} 2023-03-31 16:56:08,368 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-31 16:56:08,370 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-31 16:56:08,371 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=50, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.6470 sec 2023-03-31 16:56:08,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-31 16:56:08,834 INFO [Listener at localhost/44135] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 50 completed 2023-03-31 16:56:08,835 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-03-31 16:56:08,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure2.ProcedureExecutor(1029): Stored pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-31 16:56:08,837 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-31 16:56:08,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-03-31 16:56:08,837 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=53, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-31 16:56:08,841 DEBUG [HFileArchiver-11] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5 2023-03-31 16:56:08,842 DEBUG [HFileArchiver-11] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A, FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B, FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C, FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/recovered.edits] 2023-03-31 16:56:08,844 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/32c17e08f748497ab858c33ef63ca971 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/32c17e08f748497ab858c33ef63ca971 2023-03-31 16:56:08,845 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/efd3380443d146b19269fecbc1ebf68e to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/A/efd3380443d146b19269fecbc1ebf68e 2023-03-31 16:56:08,847 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/3f6a2533c87e40bc9c7ba49672aa0306 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/3f6a2533c87e40bc9c7ba49672aa0306 2023-03-31 16:56:08,848 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/7e4a15b332764af191b84c28a2898da8 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/B/7e4a15b332764af191b84c28a2898da8 2023-03-31 16:56:08,850 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/457610069ccc423eaa93c90cebe213df to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/457610069ccc423eaa93c90cebe213df 2023-03-31 16:56:08,851 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/69a5480eeaf745009b16284dc290eb58 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/C/69a5480eeaf745009b16284dc290eb58 2023-03-31 16:56:08,853 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/recovered.edits/1225.seqid to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5/recovered.edits/1225.seqid 2023-03-31 16:56:08,854 DEBUG [HFileArchiver-11] backup.HFileArchiver(596): Deleted hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/fd96548248d41c209e12fca5d7040af5 2023-03-31 16:56:08,854 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-31 16:56:08,856 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=53, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-31 16:56:08,857 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-31 16:56:08,858 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-31 16:56:08,859 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=53, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-31 16:56:08,859 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-31 16:56:08,859 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680281768859"}]},"ts":"9223372036854775807"} 2023-03-31 16:56:08,860 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-31 16:56:08,860 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => fd96548248d41c209e12fca5d7040af5, NAME => 'TestAcidGuarantees,,1680281742438.fd96548248d41c209e12fca5d7040af5.', STARTKEY => '', ENDKEY => ''}] 2023-03-31 16:56:08,860 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-31 16:56:08,860 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680281768860"}]},"ts":"9223372036854775807"} 2023-03-31 16:56:08,862 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-31 16:56:08,863 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=53, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-31 16:56:08,864 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=53, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 29 msec 2023-03-31 16:56:08,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-03-31 16:56:08,938 INFO [Listener at localhost/44135] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 53 completed 2023-03-31 16:56:08,950 INFO [Listener at localhost/44135] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=229 (was 237), OpenFileDescriptor=564 (was 560) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=403 (was 384) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=3952 (was 3605) - AvailableMemoryMB LEAK? - 2023-03-31 16:56:08,962 INFO [Listener at localhost/44135] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=229, OpenFileDescriptor=564, MaxFileDescriptor=60000, SystemLoadAverage=403, ProcessCount=169, AvailableMemoryMB=3951 2023-03-31 16:56:08,964 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-31 16:56:08,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-31 16:56:08,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure2.ProcedureExecutor(1029): Stored pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-31 16:56:08,967 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-31 16:56:08,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 54 2023-03-31 16:56:08,967 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-31 16:56:08,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-31 16:56:08,968 DEBUG [HFileArchiver-12] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f 2023-03-31 16:56:08,969 DEBUG [HFileArchiver-12] backup.HFileArchiver(153): Directory hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f empty. 2023-03-31 16:56:08,969 DEBUG [HFileArchiver-12] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f 2023-03-31 16:56:08,969 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-31 16:56:09,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-31 16:56:09,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-31 16:56:09,375 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-31 16:56:09,377 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 313b3df11d64a9396698e7102a97394f, NAME => 'TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp 2023-03-31 16:56:09,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-31 16:56:09,782 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-31 16:56:09,782 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 313b3df11d64a9396698e7102a97394f, disabling compactions & flushes 2023-03-31 16:56:09,782 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:09,782 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:09,782 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. after waiting 0 ms 2023-03-31 16:56:09,782 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:09,782 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:09,782 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 313b3df11d64a9396698e7102a97394f: 2023-03-31 16:56:09,784 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-31 16:56:09,785 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680281769784"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680281769784"}]},"ts":"1680281769784"} 2023-03-31 16:56:09,785 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-31 16:56:09,786 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-31 16:56:09,786 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680281769786"}]},"ts":"1680281769786"} 2023-03-31 16:56:09,787 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-31 16:56:09,790 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=313b3df11d64a9396698e7102a97394f, ASSIGN}] 2023-03-31 16:56:09,792 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=313b3df11d64a9396698e7102a97394f, ASSIGN 2023-03-31 16:56:09,792 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=313b3df11d64a9396698e7102a97394f, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,33441,1680281619375; forceNewPlan=false, retain=false 2023-03-31 16:56:09,944 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=313b3df11d64a9396698e7102a97394f, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:09,944 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680281769944"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680281769944"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680281769944"}]},"ts":"1680281769944"} 2023-03-31 16:56:09,945 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=56, ppid=55, state=RUNNABLE; OpenRegionProcedure 313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375}] 2023-03-31 16:56:10,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-31 16:56:10,100 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:10,100 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 313b3df11d64a9396698e7102a97394f, NAME => 'TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.', STARTKEY => '', ENDKEY => ''} 2023-03-31 16:56:10,100 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 313b3df11d64a9396698e7102a97394f 2023-03-31 16:56:10,100 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-31 16:56:10,100 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 313b3df11d64a9396698e7102a97394f 2023-03-31 16:56:10,100 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 313b3df11d64a9396698e7102a97394f 2023-03-31 16:56:10,102 INFO [StoreOpener-313b3df11d64a9396698e7102a97394f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 313b3df11d64a9396698e7102a97394f 2023-03-31 16:56:10,103 DEBUG [StoreOpener-313b3df11d64a9396698e7102a97394f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A 2023-03-31 16:56:10,103 DEBUG [StoreOpener-313b3df11d64a9396698e7102a97394f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A 2023-03-31 16:56:10,103 INFO [StoreOpener-313b3df11d64a9396698e7102a97394f-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-31 16:56:10,103 INFO [StoreOpener-313b3df11d64a9396698e7102a97394f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 313b3df11d64a9396698e7102a97394f columnFamilyName A 2023-03-31 16:56:10,104 INFO [StoreOpener-313b3df11d64a9396698e7102a97394f-1] regionserver.HStore(310): Store=313b3df11d64a9396698e7102a97394f/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-31 16:56:10,104 INFO [StoreOpener-313b3df11d64a9396698e7102a97394f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 313b3df11d64a9396698e7102a97394f 2023-03-31 16:56:10,105 DEBUG [StoreOpener-313b3df11d64a9396698e7102a97394f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B 2023-03-31 16:56:10,105 DEBUG [StoreOpener-313b3df11d64a9396698e7102a97394f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B 2023-03-31 16:56:10,105 INFO [StoreOpener-313b3df11d64a9396698e7102a97394f-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-31 16:56:10,105 INFO [StoreOpener-313b3df11d64a9396698e7102a97394f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 313b3df11d64a9396698e7102a97394f columnFamilyName B 2023-03-31 16:56:10,106 INFO [StoreOpener-313b3df11d64a9396698e7102a97394f-1] regionserver.HStore(310): Store=313b3df11d64a9396698e7102a97394f/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-31 16:56:10,106 INFO [StoreOpener-313b3df11d64a9396698e7102a97394f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 313b3df11d64a9396698e7102a97394f 2023-03-31 16:56:10,107 DEBUG [StoreOpener-313b3df11d64a9396698e7102a97394f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C 2023-03-31 16:56:10,107 DEBUG [StoreOpener-313b3df11d64a9396698e7102a97394f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C 2023-03-31 16:56:10,107 INFO [StoreOpener-313b3df11d64a9396698e7102a97394f-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-31 16:56:10,107 INFO [StoreOpener-313b3df11d64a9396698e7102a97394f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 313b3df11d64a9396698e7102a97394f columnFamilyName C 2023-03-31 16:56:10,107 INFO [StoreOpener-313b3df11d64a9396698e7102a97394f-1] regionserver.HStore(310): Store=313b3df11d64a9396698e7102a97394f/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-31 16:56:10,108 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:10,108 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f 2023-03-31 16:56:10,108 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f 2023-03-31 16:56:10,110 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-31 16:56:10,111 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 313b3df11d64a9396698e7102a97394f 2023-03-31 16:56:10,112 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-31 16:56:10,112 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 313b3df11d64a9396698e7102a97394f; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73902954, jitterRate=0.10123983025550842}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-31 16:56:10,112 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 313b3df11d64a9396698e7102a97394f: 2023-03-31 16:56:10,113 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f., pid=56, masterSystemTime=1680281770097 2023-03-31 16:56:10,114 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:10,114 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:10,114 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=313b3df11d64a9396698e7102a97394f, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:10,115 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680281770114"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680281770114"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680281770114"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680281770114"}]},"ts":"1680281770114"} 2023-03-31 16:56:10,116 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=56, resume processing ppid=55 2023-03-31 16:56:10,116 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=56, ppid=55, state=SUCCESS; OpenRegionProcedure 313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 in 170 msec 2023-03-31 16:56:10,118 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=55, resume processing ppid=54 2023-03-31 16:56:10,118 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=54, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=313b3df11d64a9396698e7102a97394f, ASSIGN in 326 msec 2023-03-31 16:56:10,118 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-31 16:56:10,118 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680281770118"}]},"ts":"1680281770118"} 2023-03-31 16:56:10,119 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-31 16:56:10,121 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-31 16:56:10,121 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=54, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1560 sec 2023-03-31 16:56:11,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-31 16:56:11,072 INFO [Listener at localhost/44135] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 54 completed 2023-03-31 16:56:11,073 DEBUG [Listener at localhost/44135] zookeeper.ReadOnlyZKClient(139): Connect 0x59185ba7 to 127.0.0.1:51916 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 16:56:11,078 DEBUG [Listener at localhost/44135] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6af74c0e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 16:56:11,079 DEBUG [Listener at localhost/44135] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 16:56:11,081 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49560, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 16:56:11,082 DEBUG [Listener at localhost/44135] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-31 16:56:11,084 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40244, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-31 16:56:11,086 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-31 16:56:11,086 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-31 16:56:11,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure2.ProcedureExecutor(1029): Stored pid=57, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-03-31 16:56:11,501 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-31 16:56:11,501 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-31 16:56:11,502 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-31 16:56:11,503 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=57, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-03-31 16:56:11,505 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=313b3df11d64a9396698e7102a97394f, REOPEN/MOVE}] 2023-03-31 16:56:11,506 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=313b3df11d64a9396698e7102a97394f, REOPEN/MOVE 2023-03-31 16:56:11,506 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=313b3df11d64a9396698e7102a97394f, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:11,507 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680281771506"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680281771506"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680281771506"}]},"ts":"1680281771506"} 2023-03-31 16:56:11,507 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=60, ppid=59, state=RUNNABLE; CloseRegionProcedure 313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375}] 2023-03-31 16:56:11,659 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 313b3df11d64a9396698e7102a97394f 2023-03-31 16:56:11,660 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 313b3df11d64a9396698e7102a97394f, disabling compactions & flushes 2023-03-31 16:56:11,660 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:11,660 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:11,660 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. after waiting 0 ms 2023-03-31 16:56:11,660 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:11,663 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-03-31 16:56:11,664 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:11,664 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 313b3df11d64a9396698e7102a97394f: 2023-03-31 16:56:11,664 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 313b3df11d64a9396698e7102a97394f to self. 2023-03-31 16:56:11,665 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 313b3df11d64a9396698e7102a97394f 2023-03-31 16:56:11,665 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=313b3df11d64a9396698e7102a97394f, regionState=CLOSED 2023-03-31 16:56:11,665 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680281771665"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680281771665"}]},"ts":"1680281771665"} 2023-03-31 16:56:11,667 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=60, resume processing ppid=59 2023-03-31 16:56:11,667 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=60, ppid=59, state=SUCCESS; CloseRegionProcedure 313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 in 159 msec 2023-03-31 16:56:11,668 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=313b3df11d64a9396698e7102a97394f, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,33441,1680281619375; forceNewPlan=false, retain=true 2023-03-31 16:56:11,818 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=313b3df11d64a9396698e7102a97394f, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:11,819 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680281771818"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680281771818"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680281771818"}]},"ts":"1680281771818"} 2023-03-31 16:56:11,820 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=59, state=RUNNABLE; OpenRegionProcedure 313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375}] 2023-03-31 16:56:11,974 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:11,975 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 313b3df11d64a9396698e7102a97394f, NAME => 'TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.', STARTKEY => '', ENDKEY => ''} 2023-03-31 16:56:11,975 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 313b3df11d64a9396698e7102a97394f 2023-03-31 16:56:11,975 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-31 16:56:11,975 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 313b3df11d64a9396698e7102a97394f 2023-03-31 16:56:11,975 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 313b3df11d64a9396698e7102a97394f 2023-03-31 16:56:11,976 INFO [StoreOpener-313b3df11d64a9396698e7102a97394f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 313b3df11d64a9396698e7102a97394f 2023-03-31 16:56:11,977 DEBUG [StoreOpener-313b3df11d64a9396698e7102a97394f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A 2023-03-31 16:56:11,977 DEBUG [StoreOpener-313b3df11d64a9396698e7102a97394f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A 2023-03-31 16:56:11,977 INFO [StoreOpener-313b3df11d64a9396698e7102a97394f-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-31 16:56:11,977 INFO [StoreOpener-313b3df11d64a9396698e7102a97394f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 313b3df11d64a9396698e7102a97394f columnFamilyName A 2023-03-31 16:56:11,978 INFO [StoreOpener-313b3df11d64a9396698e7102a97394f-1] regionserver.HStore(310): Store=313b3df11d64a9396698e7102a97394f/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-31 16:56:11,979 INFO [StoreOpener-313b3df11d64a9396698e7102a97394f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 313b3df11d64a9396698e7102a97394f 2023-03-31 16:56:11,979 DEBUG [StoreOpener-313b3df11d64a9396698e7102a97394f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B 2023-03-31 16:56:11,979 DEBUG [StoreOpener-313b3df11d64a9396698e7102a97394f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B 2023-03-31 16:56:11,979 INFO [StoreOpener-313b3df11d64a9396698e7102a97394f-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-31 16:56:11,980 INFO [StoreOpener-313b3df11d64a9396698e7102a97394f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 313b3df11d64a9396698e7102a97394f columnFamilyName B 2023-03-31 16:56:11,980 INFO [StoreOpener-313b3df11d64a9396698e7102a97394f-1] regionserver.HStore(310): Store=313b3df11d64a9396698e7102a97394f/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-31 16:56:11,980 INFO [StoreOpener-313b3df11d64a9396698e7102a97394f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 313b3df11d64a9396698e7102a97394f 2023-03-31 16:56:11,981 DEBUG [StoreOpener-313b3df11d64a9396698e7102a97394f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C 2023-03-31 16:56:11,981 DEBUG [StoreOpener-313b3df11d64a9396698e7102a97394f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C 2023-03-31 16:56:11,981 INFO [StoreOpener-313b3df11d64a9396698e7102a97394f-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-31 16:56:11,981 INFO [StoreOpener-313b3df11d64a9396698e7102a97394f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 313b3df11d64a9396698e7102a97394f columnFamilyName C 2023-03-31 16:56:11,982 INFO [StoreOpener-313b3df11d64a9396698e7102a97394f-1] regionserver.HStore(310): Store=313b3df11d64a9396698e7102a97394f/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-31 16:56:11,982 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:11,982 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f 2023-03-31 16:56:11,983 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f 2023-03-31 16:56:11,984 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-31 16:56:11,985 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 313b3df11d64a9396698e7102a97394f 2023-03-31 16:56:11,986 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 313b3df11d64a9396698e7102a97394f; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67002338, jitterRate=-0.0015873610973358154}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-31 16:56:11,986 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 313b3df11d64a9396698e7102a97394f: 2023-03-31 16:56:11,987 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f., pid=61, masterSystemTime=1680281771972 2023-03-31 16:56:11,988 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:11,988 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:11,988 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=313b3df11d64a9396698e7102a97394f, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:11,988 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680281771988"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680281771988"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680281771988"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680281771988"}]},"ts":"1680281771988"} 2023-03-31 16:56:11,990 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=59 2023-03-31 16:56:11,990 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=59, state=SUCCESS; OpenRegionProcedure 313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 in 169 msec 2023-03-31 16:56:11,991 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=59, resume processing ppid=58 2023-03-31 16:56:11,991 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=58, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=313b3df11d64a9396698e7102a97394f, REOPEN/MOVE in 485 msec 2023-03-31 16:56:11,993 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=57 2023-03-31 16:56:11,993 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=57, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 488 msec 2023-03-31 16:56:11,994 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=57, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 907 msec 2023-03-31 16:56:11,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1227): Checking to see if procedure is done pid=57 2023-03-31 16:56:11,995 DEBUG [Listener at localhost/44135] zookeeper.ReadOnlyZKClient(139): Connect 0x062b3caa to 127.0.0.1:51916 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 16:56:12,000 DEBUG [Listener at localhost/44135] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1d759aab, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 16:56:12,001 DEBUG [Listener at localhost/44135] zookeeper.ReadOnlyZKClient(139): Connect 0x66b5b1f3 to 127.0.0.1:51916 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 16:56:12,005 DEBUG [Listener at localhost/44135] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@39241fe3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 16:56:12,006 DEBUG [Listener at localhost/44135] zookeeper.ReadOnlyZKClient(139): Connect 0x544a4215 to 127.0.0.1:51916 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 16:56:12,010 DEBUG [Listener at localhost/44135] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1fbb5b46, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 16:56:12,010 DEBUG [Listener at localhost/44135] zookeeper.ReadOnlyZKClient(139): Connect 0x25baee3e to 127.0.0.1:51916 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 16:56:12,015 DEBUG [Listener at localhost/44135] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@23e65a76, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 16:56:12,015 DEBUG [Listener at localhost/44135] zookeeper.ReadOnlyZKClient(139): Connect 0x2c53bf96 to 127.0.0.1:51916 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 16:56:12,018 DEBUG [Listener at localhost/44135] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5e6a6f3c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 16:56:12,019 DEBUG [Listener at localhost/44135] zookeeper.ReadOnlyZKClient(139): Connect 0x0a470a3d to 127.0.0.1:51916 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 16:56:12,027 DEBUG [Listener at localhost/44135] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7d997c6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 16:56:12,027 DEBUG [Listener at localhost/44135] zookeeper.ReadOnlyZKClient(139): Connect 0x4bb72437 to 127.0.0.1:51916 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 16:56:12,037 DEBUG [Listener at localhost/44135] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7993d99e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 16:56:12,038 DEBUG [Listener at localhost/44135] zookeeper.ReadOnlyZKClient(139): Connect 0x3dca23c0 to 127.0.0.1:51916 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 16:56:12,045 DEBUG [Listener at localhost/44135] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4fae3ed2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 16:56:12,045 DEBUG [Listener at localhost/44135] zookeeper.ReadOnlyZKClient(139): Connect 0x62d52321 to 127.0.0.1:51916 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 16:56:12,050 DEBUG [Listener at localhost/44135] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@eedd898, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 16:56:12,050 DEBUG [Listener at localhost/44135] zookeeper.ReadOnlyZKClient(139): Connect 0x2a8bf279 to 127.0.0.1:51916 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-31 16:56:12,056 DEBUG [Listener at localhost/44135] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@618c0022, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-31 16:56:12,067 DEBUG [hconnection-0x266bc00-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 16:56:12,067 DEBUG [hconnection-0x7e4ef054-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 16:56:12,068 DEBUG [hconnection-0x6d78a6ff-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 16:56:12,068 DEBUG [hconnection-0x5cdf9472-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 16:56:12,069 DEBUG [hconnection-0x61bec753-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 16:56:12,073 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49568, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 16:56:12,069 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49584, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 16:56:12,074 DEBUG [hconnection-0x4ae7f7f1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 16:56:12,074 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:12,074 DEBUG [hconnection-0x76146ff1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 16:56:12,073 DEBUG [hconnection-0x13fbea4d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 16:56:12,073 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49566, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 16:56:12,075 DEBUG [hconnection-0x1f9bd000-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 16:56:12,077 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49602, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 16:56:12,077 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49606, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 16:56:12,078 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49634, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 16:56:12,080 DEBUG [hconnection-0x694ffec5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-31 16:56:12,080 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49600, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 16:56:12,081 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49618, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 16:56:12,081 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49608, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 16:56:12,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:12,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:12,082 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:12,082 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:12,082 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:12,084 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:12,087 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:12,087 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:12,087 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49640, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-31 16:56:12,089 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,089 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:12,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:12,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:12,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:12,090 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,091 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:12,091 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:12,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:12,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:12,092 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:12,096 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:12,097 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:12,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on 313b3df11d64a9396698e7102a97394f 2023-03-31 16:56:12,097 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 313b3df11d64a9396698e7102a97394f 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-31 16:56:12,097 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:12,097 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:12,097 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:12,098 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=A 2023-03-31 16:56:12,098 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:56:12,098 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=B 2023-03-31 16:56:12,098 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:56:12,098 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=C 2023-03-31 16:56:12,098 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:56:12,100 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:12,100 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:12,100 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:12,100 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. started... 2023-03-31 16:56:12,100 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. as already flushing 2023-03-31 16:56:12,100 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:12,100 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:12,101 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:12,101 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,101 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:12,101 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:12,103 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,103 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:12,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:12,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:12,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:12,104 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:12,104 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:12,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:12,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:12,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:12,105 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:12,105 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69b7fc64[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:12,105 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:12,105 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:12,106 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:12,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:12,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:12,107 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:12,107 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,107 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:12,107 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:12,107 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:12,113 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:12,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49602 deadline: 1680281832111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,114 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:12,114 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,114 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:12,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:12,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:12,114 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:12,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:12,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:12,115 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:12,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:12,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49618 deadline: 1680281832112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,115 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:12,115 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:12,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49568 deadline: 1680281832112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49566 deadline: 1680281832112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,116 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:12,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49608 deadline: 1680281832113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:12,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:12,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:12,117 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:12,117 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:12,117 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:12,117 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:12,117 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:12,117 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:12,119 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:12,119 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:12,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:12,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:12,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:12,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:12,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:12,119 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:12,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:12,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:12,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:12,119 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:12,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:12,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:12,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:12,121 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:12,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:12,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,123 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:12,123 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:12,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:12,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:12,123 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:12,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:12,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:12,123 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,123 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:12,123 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:12,123 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,123 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:12,123 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:12,123 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:12,123 DEBUG [Thread-4140] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:12,123 DEBUG [Thread-4140] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:12,123 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:56:12,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:12,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:12,215 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:12,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49602 deadline: 1680281832214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,218 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:12,218 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:12,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49618 deadline: 1680281832216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,218 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:12,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49566 deadline: 1680281832216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,218 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:12,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49568 deadline: 1680281832216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49608 deadline: 1680281832217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,223 DEBUG [Thread-4140] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:12,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:12,225 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:12,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:12,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:12,227 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:12,227 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:12,227 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:12,227 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:12,228 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:12,228 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:12,229 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:12,229 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:12,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:12,230 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:12,230 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:12,230 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:12,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:12,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:12,230 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:12,231 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:12,231 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:12,231 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:12,231 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:12,231 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:12,231 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:12,231 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:12,231 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:12,231 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. started... 2023-03-31 16:56:12,232 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. as already flushing 2023-03-31 16:56:12,232 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:12,232 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:12,232 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:12,232 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,232 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:12,232 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:12,234 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,234 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:12,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:12,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:12,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:12,234 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:12,235 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:12,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:12,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:12,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:12,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:12,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2fdd894f[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:12,236 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:12,236 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:12,240 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:12,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:12,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:12,240 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,240 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:12,240 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:12,240 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:12,240 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:12,242 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:12,242 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:12,242 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,242 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:12,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:12,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:12,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:12,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:12,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:12,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:12,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:12,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:12,245 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:12,245 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:12,245 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:12,245 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:12,245 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:12,245 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:12,248 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:12,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:12,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:12,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:12,248 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:12,248 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:12,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:12,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:12,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:12,248 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:12,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:12,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:12,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:12,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:12,250 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:12,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:12,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:12,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,253 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:12,253 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,253 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:12,253 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:12,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:12,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:12,253 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:12,253 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:12,253 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,253 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:12,253 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:12,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:12,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:12,253 DEBUG [Thread-4140] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:12,253 DEBUG [Thread-4140] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:12,253 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:12,253 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:56:12,254 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:12,254 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:12,354 DEBUG [Thread-4140] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:12,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:12,356 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:12,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:12,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:12,357 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:12,357 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:12,357 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:12,358 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:12,358 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:12,358 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:12,360 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:12,360 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:12,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:12,360 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,360 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:12,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:12,360 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:12,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:12,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:12,361 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:12,361 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:12,361 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:12,361 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:12,361 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:12,361 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:12,362 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:12,362 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:12,362 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. started... 2023-03-31 16:56:12,362 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. as already flushing 2023-03-31 16:56:12,362 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:12,362 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:12,362 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:12,362 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,362 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:12,362 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:12,365 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,365 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:12,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:12,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:12,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:12,365 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:12,366 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:12,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:12,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:12,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:12,368 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:12,368 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4284de16[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:12,368 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:12,368 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:12,370 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:12,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:12,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:12,371 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:12,371 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,371 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:12,371 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:12,371 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:12,372 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,372 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:12,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:12,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:12,372 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:12,373 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:12,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:12,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:12,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:12,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:12,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:12,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:12,375 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:12,375 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:12,375 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:12,375 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:12,375 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:12,375 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:12,377 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:12,377 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:12,377 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:12,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:12,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:12,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:12,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:12,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:12,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:12,378 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:12,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:12,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:12,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:12,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:12,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:12,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,379 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:12,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:12,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,382 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:12,382 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:12,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:12,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:12,382 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,382 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:12,382 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:12,382 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,382 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:12,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:12,382 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:12,382 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:12,382 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:56:12,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:12,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:12,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:12,382 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:12,383 DEBUG [Thread-4140] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:12,383 DEBUG [Thread-4140] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:12,417 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:12,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49602 deadline: 1680281832416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,420 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:12,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49618 deadline: 1680281832419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,421 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:12,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49608 deadline: 1680281832419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,421 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:12,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49566 deadline: 1680281832419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,421 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:12,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49568 deadline: 1680281832420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,483 DEBUG [Thread-4140] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:12,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:12,485 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:12,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:12,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:12,487 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:12,487 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:12,487 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:12,487 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:12,487 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:12,487 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:12,489 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:12,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:12,489 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:12,490 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,490 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:12,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:12,490 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:12,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:12,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:12,491 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:12,491 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:12,491 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:12,491 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:12,491 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:12,491 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:12,491 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:12,491 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:12,491 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:12,491 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. started... 2023-03-31 16:56:12,491 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. as already flushing 2023-03-31 16:56:12,491 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:12,491 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:12,492 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:12,492 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,492 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:12,492 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:12,493 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:12,493 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,493 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:12,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:12,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:12,493 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:12,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:12,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:12,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:12,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:12,495 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:12,495 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e3f3f74[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:12,495 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:12,495 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:12,496 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:12,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:12,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:12,497 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:12,497 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,497 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:12,497 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:12,497 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:12,498 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,498 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:12,498 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:12,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:12,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:12,498 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:12,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:12,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:12,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:12,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:12,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:12,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:12,501 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:12,501 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:12,501 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:12,501 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:12,501 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:12,501 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:12,504 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:12,504 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:12,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:12,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:12,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:12,504 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:12,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:12,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:12,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:12,504 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:12,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:12,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:12,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:12,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:12,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:12,506 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:12,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:12,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,508 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,508 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:12,508 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:12,508 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:12,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:12,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:12,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:12,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:12,508 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:12,508 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,508 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:12,509 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:12,509 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:12,508 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:12,508 DEBUG [Thread-4140] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:12,509 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:56:12,509 DEBUG [Thread-4140] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:12,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:12,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:12,518 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230331b52882b9355b474196a2371681394258 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331b52882b9355b474196a2371681394258 2023-03-31 16:56:12,522 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=17, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/A/eb5b7a0d6a874c09a9cbda570e57438e 2023-03-31 16:56:12,544 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=17 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/B/c59fa43c80e84d598d160426cd7f7e11 2023-03-31 16:56:12,562 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=17 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/C/5a8ff4c9182740e6927a7a1dd5b32dde 2023-03-31 16:56:12,566 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/A/eb5b7a0d6a874c09a9cbda570e57438e as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/eb5b7a0d6a874c09a9cbda570e57438e 2023-03-31 16:56:12,569 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/eb5b7a0d6a874c09a9cbda570e57438e, entries=200, sequenceid=17, filesize=31.9 K 2023-03-31 16:56:12,570 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/B/c59fa43c80e84d598d160426cd7f7e11 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/c59fa43c80e84d598d160426cd7f7e11 2023-03-31 16:56:12,573 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/c59fa43c80e84d598d160426cd7f7e11, entries=150, sequenceid=17, filesize=11.6 K 2023-03-31 16:56:12,574 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/C/5a8ff4c9182740e6927a7a1dd5b32dde as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/5a8ff4c9182740e6927a7a1dd5b32dde 2023-03-31 16:56:12,577 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/5a8ff4c9182740e6927a7a1dd5b32dde, entries=150, sequenceid=17, filesize=11.6 K 2023-03-31 16:56:12,577 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 313b3df11d64a9396698e7102a97394f in 480ms, sequenceid=17, compaction requested=false 2023-03-31 16:56:12,577 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 313b3df11d64a9396698e7102a97394f: 2023-03-31 16:56:12,609 DEBUG [Thread-4140] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:12,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:12,611 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:12,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:12,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:12,612 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:12,612 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:12,613 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:12,613 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:12,613 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:12,613 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:12,614 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:12,614 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:12,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:12,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:12,615 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,615 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:12,615 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:12,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:12,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:12,616 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:12,616 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:12,616 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:12,616 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:12,616 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:12,617 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:12,617 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:12,617 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:12,617 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:12,617 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. started... 2023-03-31 16:56:12,617 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 313b3df11d64a9396698e7102a97394f 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-31 16:56:12,617 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=A 2023-03-31 16:56:12,617 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:56:12,617 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=B 2023-03-31 16:56:12,617 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:56:12,618 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=C 2023-03-31 16:56:12,618 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:56:12,633 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023033184aa2f97bcc54b978542188fbf4b0444 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033184aa2f97bcc54b978542188fbf4b0444 2023-03-31 16:56:12,653 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=36, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/A/d3673ed8fc5545708d85b0af48398a9f 2023-03-31 16:56:12,663 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/B/eb3aaf9091114180bdfb312b7b548ff5 2023-03-31 16:56:12,674 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/C/ec02caf3bdaf4027a5bd1ab53d2d9c0c 2023-03-31 16:56:12,677 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/A/d3673ed8fc5545708d85b0af48398a9f as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/d3673ed8fc5545708d85b0af48398a9f 2023-03-31 16:56:12,681 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/d3673ed8fc5545708d85b0af48398a9f, entries=150, sequenceid=36, filesize=25.1 K 2023-03-31 16:56:12,681 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/B/eb3aaf9091114180bdfb312b7b548ff5 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/eb3aaf9091114180bdfb312b7b548ff5 2023-03-31 16:56:12,684 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/eb3aaf9091114180bdfb312b7b548ff5, entries=150, sequenceid=36, filesize=11.6 K 2023-03-31 16:56:12,685 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/C/ec02caf3bdaf4027a5bd1ab53d2d9c0c as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/ec02caf3bdaf4027a5bd1ab53d2d9c0c 2023-03-31 16:56:12,689 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/ec02caf3bdaf4027a5bd1ab53d2d9c0c, entries=150, sequenceid=36, filesize=11.6 K 2023-03-31 16:56:12,689 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=0 B/0 for 313b3df11d64a9396698e7102a97394f in 72ms, sequenceid=36, compaction requested=false 2023-03-31 16:56:12,689 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 313b3df11d64a9396698e7102a97394f: 2023-03-31 16:56:12,689 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:12,689 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:12,689 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:12,689 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,690 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:12,690 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:12,691 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,691 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:12,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:12,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:12,692 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:12,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:12,692 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:12,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:12,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:12,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:12,693 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:12,693 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48bdb40b[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:12,693 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:12,693 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:12,695 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:12,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:12,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:12,696 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:12,696 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:12,696 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:12,696 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,696 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:12,697 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,697 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:12,697 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:12,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:12,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:12,697 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:12,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:12,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:12,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:12,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:12,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:12,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:12,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:12,700 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:12,700 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:12,700 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:12,700 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:12,700 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:12,704 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:12,704 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:12,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:12,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:12,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:12,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:12,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:12,704 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:12,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:12,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:12,705 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:12,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:12,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:12,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:12,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:12,708 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:12,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:12,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,710 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,710 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:12,710 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:12,710 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:56:12,710 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:12,711 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:12,710 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:12,711 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:12,710 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:12,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:12,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:12,711 DEBUG [Thread-4140] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:12,711 DEBUG [Thread-4140] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:12,711 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,711 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:12,711 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:12,711 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:12,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:12,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:12,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on 313b3df11d64a9396698e7102a97394f 2023-03-31 16:56:12,734 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 313b3df11d64a9396698e7102a97394f 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-31 16:56:12,735 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=A 2023-03-31 16:56:12,735 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:56:12,735 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=B 2023-03-31 16:56:12,735 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:56:12,735 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=C 2023-03-31 16:56:12,735 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:56:12,756 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230331694841752f9d4c9c8a1f4e49c8ccc251 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331694841752f9d4c9c8a1f4e49c8ccc251 2023-03-31 16:56:12,762 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680281772753, max timestamp=1680281772760Num uniques 859; 2023-03-31 16:56:12,762 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680281772753, max timestamp=1680281772760Num uniques 859; 2023-03-31 16:56:12,762 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:12,762 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 16:56:12,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49566 deadline: 1680281832758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,762 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-31 16:56:12,764 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680281772757, max timestamp=1680281772760Num uniques 859; 2023-03-31 16:56:12,764 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-03-31 16:56:12,766 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=47, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/A/39319352866046f294566564c5b8cdea 2023-03-31 16:56:12,786 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:12,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49568 deadline: 1680281832781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,788 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:12,788 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:12,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49618 deadline: 1680281832785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49608 deadline: 1680281832786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,789 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:12,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49602 deadline: 1680281832786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,811 DEBUG [Thread-4140] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:12,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:12,813 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:12,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:12,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:12,814 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:12,814 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:12,815 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:12,815 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:12,815 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:12,815 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:12,817 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:12,817 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:12,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:12,818 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,818 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:12,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:12,818 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:12,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:12,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:12,818 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:12,819 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:12,819 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:12,819 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:12,819 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:12,819 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:12,819 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:12,819 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:12,819 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:12,819 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. started... 2023-03-31 16:56:12,819 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. as already flushing 2023-03-31 16:56:12,819 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:12,819 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:12,819 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:12,819 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,819 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:12,819 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:12,821 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:12,821 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:12,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:12,821 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:12,821 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:12,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:12,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:12,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:12,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:12,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:12,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b69cba2[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:12,822 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:12,822 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:12,824 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:12,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:12,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:12,824 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:12,824 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:12,824 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:12,824 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,824 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:12,825 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,825 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:12,825 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:12,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:12,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:12,825 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:12,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:12,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:12,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:12,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:12,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:12,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:12,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:12,828 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:12,828 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:12,828 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:12,828 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:12,828 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:12,835 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:12,835 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:12,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:12,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:12,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:12,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:12,835 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:12,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:12,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:12,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:12,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:12,836 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:12,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:12,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:12,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:12,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,837 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:12,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:12,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,840 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:12,840 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:12,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:12,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:12,840 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,840 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:12,840 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:12,840 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,840 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:12,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:12,840 DEBUG [Thread-4140] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:12,840 DEBUG [Thread-4140] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:12,840 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:12,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:12,840 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:12,840 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:12,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:12,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:12,867 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:12,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49566 deadline: 1680281832864, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,889 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:12,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49568 deadline: 1680281832887, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,893 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:12,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49618 deadline: 1680281832890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,893 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:12,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49608 deadline: 1680281832890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,893 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:12,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49602 deadline: 1680281832890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,940 DEBUG [Thread-4140] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:12,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:12,942 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:12,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:12,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:12,944 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:12,944 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:12,944 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:12,944 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:12,944 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:12,944 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:12,945 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:12,945 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:12,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:12,946 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,946 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:12,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:12,946 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:12,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:12,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:12,946 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:12,947 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:12,947 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:12,947 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:12,947 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:12,947 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:12,947 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:12,947 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:12,947 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:12,947 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. started... 2023-03-31 16:56:12,947 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. as already flushing 2023-03-31 16:56:12,947 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:12,947 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:12,947 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:12,947 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,947 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:12,947 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:12,949 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:12,949 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:12,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:12,949 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:12,949 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:12,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:12,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:12,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:12,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:12,951 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:12,951 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ff9c0b7[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:12,951 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:12,951 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:12,953 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:12,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:12,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:12,953 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:12,953 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:12,953 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:12,953 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,953 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:12,955 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,955 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:12,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:12,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:12,955 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:12,955 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:12,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:12,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:12,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:12,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:12,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:12,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:12,957 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:12,957 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:12,957 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:12,957 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:12,957 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:12,957 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:12,958 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:12,958 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:12,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:12,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:12,959 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:12,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:12,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:12,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:12,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:12,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:12,959 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:12,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:12,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:12,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:12,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:12,960 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:12,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:12,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,962 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,962 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:12,962 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:12,962 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:12,963 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:12,962 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:12,963 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:12,963 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:12,963 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:12,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:12,963 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:12,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:12,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:12,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:12,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:12,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:12,963 DEBUG [Thread-4140] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:12,963 DEBUG [Thread-4140] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:12,963 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:56:13,063 DEBUG [Thread-4140] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:13,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:13,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:13,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:13,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:13,067 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:13,067 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:13,067 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:13,067 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:13,068 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:13,068 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:13,070 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:13,070 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:13,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:13,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:13,070 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49566 deadline: 1680281833068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:13,071 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,071 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:13,071 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:13,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:13,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:13,071 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:13,072 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:13,072 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:13,072 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:13,072 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:13,072 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:13,072 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:13,072 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:13,072 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:13,072 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. started... 2023-03-31 16:56:13,072 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. as already flushing 2023-03-31 16:56:13,072 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:13,072 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:13,072 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:13,072 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,072 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:13,072 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:13,074 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,074 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:13,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:13,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:13,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:13,074 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:13,074 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:13,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:13,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:13,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:13,075 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:13,076 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8da96a3[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:13,076 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:13,076 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:13,077 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:13,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:13,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:13,077 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:13,077 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:13,077 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:13,077 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,077 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:13,078 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,078 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:13,078 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:13,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:13,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:13,078 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:13,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:13,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:13,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:13,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:13,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:13,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:13,081 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:13,081 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:13,081 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:13,081 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:13,081 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:13,081 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:13,083 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:13,083 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:13,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:13,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:13,083 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:13,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:13,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:13,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:13,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:13,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:13,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:13,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:13,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:13,085 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:13,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:13,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:13,086 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:13,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,091 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,091 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:13,091 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:13,091 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:13,091 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:13,091 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:13,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:13,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:13,091 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:13,091 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:13,092 DEBUG [Thread-4140] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:13,092 DEBUG [Thread-4140] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:13,091 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,092 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:13,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:13,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:13,092 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:13,092 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:13,092 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:13,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49568 deadline: 1680281833090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,096 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:13,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49608 deadline: 1680281833094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,097 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:13,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49602 deadline: 1680281833094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,097 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:13,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49618 deadline: 1680281833095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,186 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=47 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/B/a962a4ad2d8f4dbcaf6fed629f966107 2023-03-31 16:56:13,192 DEBUG [Thread-4140] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:13,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:13,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:13,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:13,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:13,196 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:13,196 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:13,196 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:13,196 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:13,196 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=47 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/C/f93eff84a48f4b20ab9b211d2a010793 2023-03-31 16:56:13,196 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:13,196 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:13,198 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:13,198 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:13,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:13,198 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,198 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:13,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:13,199 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:13,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:13,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:13,199 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:13,200 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:13,200 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:13,200 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:13,200 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:13,200 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:13,200 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:13,200 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:13,200 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:13,200 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. started... 2023-03-31 16:56:13,201 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. as already flushing 2023-03-31 16:56:13,201 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:13,201 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:13,201 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:13,201 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,201 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:13,201 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:13,201 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/A/39319352866046f294566564c5b8cdea as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/39319352866046f294566564c5b8cdea 2023-03-31 16:56:13,203 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,203 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:13,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:13,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:13,204 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:13,204 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:13,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:13,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:13,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:13,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:13,206 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:13,206 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2de2765f[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:13,206 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:13,206 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:13,208 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/39319352866046f294566564c5b8cdea, entries=400, sequenceid=47, filesize=59.0 K 2023-03-31 16:56:13,208 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:13,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:13,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:13,208 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:13,208 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:13,208 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:13,209 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,209 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:13,209 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/B/a962a4ad2d8f4dbcaf6fed629f966107 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/a962a4ad2d8f4dbcaf6fed629f966107 2023-03-31 16:56:13,210 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:13,210 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:13,210 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:13,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:13,210 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:13,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:13,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:13,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:13,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:13,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:13,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,213 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/a962a4ad2d8f4dbcaf6fed629f966107, entries=150, sequenceid=47, filesize=11.6 K 2023-03-31 16:56:13,214 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/C/f93eff84a48f4b20ab9b211d2a010793 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/f93eff84a48f4b20ab9b211d2a010793 2023-03-31 16:56:13,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:13,214 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:13,214 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:13,214 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:13,214 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:13,214 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:13,214 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:13,217 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:13,217 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,217 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:13,217 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:13,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:13,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:13,217 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/f93eff84a48f4b20ab9b211d2a010793, entries=150, sequenceid=47, filesize=11.6 K 2023-03-31 16:56:13,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:13,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:13,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:13,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:13,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:13,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:13,218 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:13,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:13,218 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=136.02 KB/139285 for 313b3df11d64a9396698e7102a97394f in 484ms, sequenceid=47, compaction requested=true 2023-03-31 16:56:13,218 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 313b3df11d64a9396698e7102a97394f: 2023-03-31 16:56:13,218 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 16:56:13,218 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:56:13,218 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:56:13,219 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 16:56:13,219 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:56:13,219 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 118886 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 16:56:13,219 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1912): 313b3df11d64a9396698e7102a97394f/A is initiating minor compaction (all files) 2023-03-31 16:56:13,219 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 16:56:13,219 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 313b3df11d64a9396698e7102a97394f/A in TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:13,219 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): 313b3df11d64a9396698e7102a97394f/B is initiating minor compaction (all files) 2023-03-31 16:56:13,219 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/eb5b7a0d6a874c09a9cbda570e57438e, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/d3673ed8fc5545708d85b0af48398a9f, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/39319352866046f294566564c5b8cdea] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp, totalSize=116.1 K 2023-03-31 16:56:13,219 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 313b3df11d64a9396698e7102a97394f/B in TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:13,220 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/c59fa43c80e84d598d160426cd7f7e11, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/eb3aaf9091114180bdfb312b7b548ff5, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/a962a4ad2d8f4dbcaf6fed629f966107] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp, totalSize=34.8 K 2023-03-31 16:56:13,220 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting eb5b7a0d6a874c09a9cbda570e57438e, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1680281772092 2023-03-31 16:56:13,220 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting c59fa43c80e84d598d160426cd7f7e11, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1680281772094 2023-03-31 16:56:13,220 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting d3673ed8fc5545708d85b0af48398a9f, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1680281772110 2023-03-31 16:56:13,220 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting eb3aaf9091114180bdfb312b7b548ff5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1680281772110 2023-03-31 16:56:13,221 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 39319352866046f294566564c5b8cdea, keycount=400, bloomtype=ROW, size=59.0 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1680281772725 2023-03-31 16:56:13,221 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting a962a4ad2d8f4dbcaf6fed629f966107, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1680281772728 2023-03-31 16:56:13,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:13,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:13,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,226 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): 313b3df11d64a9396698e7102a97394f#B#compaction#569 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:56:13,229 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,229 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:13,229 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:13,229 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:13,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:13,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:13,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:13,229 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:13,229 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:13,229 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:13,229 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,229 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:13,229 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:13,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:13,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:13,229 DEBUG [Thread-4140] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:13,229 DEBUG [Thread-4140] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:13,229 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:13,229 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:13,230 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:56:13,242 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] throttle.PressureAwareThroughputController(145): 313b3df11d64a9396698e7102a97394f#A#compaction#570 average throughput is 1.23 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:56:13,246 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/B/58787737611d4d6fa2826a94387010f7 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/58787737611d4d6fa2826a94387010f7 2023-03-31 16:56:13,257 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 313b3df11d64a9396698e7102a97394f/B of 313b3df11d64a9396698e7102a97394f into 58787737611d4d6fa2826a94387010f7(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:56:13,257 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 313b3df11d64a9396698e7102a97394f: 2023-03-31 16:56:13,257 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f., storeName=313b3df11d64a9396698e7102a97394f/B, priority=13, startTime=1680281773218; duration=0sec 2023-03-31 16:56:13,257 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:56:13,257 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 16:56:13,258 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 16:56:13,258 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): 313b3df11d64a9396698e7102a97394f/C is initiating minor compaction (all files) 2023-03-31 16:56:13,258 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 313b3df11d64a9396698e7102a97394f/C in TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:13,258 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/5a8ff4c9182740e6927a7a1dd5b32dde, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/ec02caf3bdaf4027a5bd1ab53d2d9c0c, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/f93eff84a48f4b20ab9b211d2a010793] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp, totalSize=34.8 K 2023-03-31 16:56:13,258 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 5a8ff4c9182740e6927a7a1dd5b32dde, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1680281772094 2023-03-31 16:56:13,259 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting ec02caf3bdaf4027a5bd1ab53d2d9c0c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1680281772110 2023-03-31 16:56:13,259 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting f93eff84a48f4b20ab9b211d2a010793, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1680281772728 2023-03-31 16:56:13,266 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): 313b3df11d64a9396698e7102a97394f#C#compaction#571 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:56:13,319 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/C/dd27abe1f1c942da90188f1f69ba5ee2 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/dd27abe1f1c942da90188f1f69ba5ee2 2023-03-31 16:56:13,324 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 313b3df11d64a9396698e7102a97394f/C of 313b3df11d64a9396698e7102a97394f into dd27abe1f1c942da90188f1f69ba5ee2(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:56:13,324 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 313b3df11d64a9396698e7102a97394f: 2023-03-31 16:56:13,324 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f., storeName=313b3df11d64a9396698e7102a97394f/C, priority=13, startTime=1680281773219; duration=0sec 2023-03-31 16:56:13,324 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:56:13,329 DEBUG [Thread-4140] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:13,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:13,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:13,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:13,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:13,332 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:13,332 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:13,333 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:13,333 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:13,333 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:13,333 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:13,334 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:13,334 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:13,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:13,335 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:13,335 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:13,335 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:13,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:13,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:13,335 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:13,336 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:13,336 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:13,336 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:13,336 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:13,336 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:13,344 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:13,344 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:13,344 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:13,344 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. started... 2023-03-31 16:56:13,344 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 313b3df11d64a9396698e7102a97394f 3/3 column families, dataSize=136.02 KB heapSize=384.46 KB 2023-03-31 16:56:13,345 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=A 2023-03-31 16:56:13,345 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:56:13,345 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=B 2023-03-31 16:56:13,345 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:56:13,345 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=C 2023-03-31 16:56:13,345 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:56:13,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on 313b3df11d64a9396698e7102a97394f 2023-03-31 16:56:13,377 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. as already flushing 2023-03-31 16:56:13,380 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023033164a3e82d210e4020bb6b10269707ca4b to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033164a3e82d210e4020bb6b10269707ca4b 2023-03-31 16:56:13,399 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:13,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49602 deadline: 1680281833397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,400 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:13,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49608 deadline: 1680281833398, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,403 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:13,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49618 deadline: 1680281833399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,404 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:13,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49566 deadline: 1680281833399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,404 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:13,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49568 deadline: 1680281833399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,406 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=85, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/A/36e4511ef13f47fcb823cbd9645b2920 2023-03-31 16:56:13,425 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=85 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/B/db0e267fa452488fae44c86b816db169 2023-03-31 16:56:13,433 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.09 KB at sequenceid=85 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/C/5f9c96f314314f298418c829edf13340 2023-03-31 16:56:13,436 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/A/36e4511ef13f47fcb823cbd9645b2920 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/36e4511ef13f47fcb823cbd9645b2920 2023-03-31 16:56:13,440 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/36e4511ef13f47fcb823cbd9645b2920, entries=150, sequenceid=85, filesize=25.1 K 2023-03-31 16:56:13,441 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/B/db0e267fa452488fae44c86b816db169 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/db0e267fa452488fae44c86b816db169 2023-03-31 16:56:13,444 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/db0e267fa452488fae44c86b816db169, entries=150, sequenceid=85, filesize=11.6 K 2023-03-31 16:56:13,445 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/C/5f9c96f314314f298418c829edf13340 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/5f9c96f314314f298418c829edf13340 2023-03-31 16:56:13,448 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/5f9c96f314314f298418c829edf13340, entries=150, sequenceid=85, filesize=11.6 K 2023-03-31 16:56:13,449 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~136.02 KB/139285, heapSize ~384.41 KB/393640, currentSize=46.96 KB/48090 for 313b3df11d64a9396698e7102a97394f in 104ms, sequenceid=85, compaction requested=false 2023-03-31 16:56:13,449 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 313b3df11d64a9396698e7102a97394f: 2023-03-31 16:56:13,449 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:13,449 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:13,449 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:13,449 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,449 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:13,449 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:13,452 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,452 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:13,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:13,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:13,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:13,453 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:13,453 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:13,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:13,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:13,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:13,455 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:13,455 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68138c9a[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:13,455 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:13,455 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:13,456 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:13,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:13,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:13,456 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:13,456 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:13,456 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:13,456 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,456 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:13,457 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:13,458 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,458 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:13,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:13,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:13,458 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:13,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:13,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:13,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:13,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:13,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:13,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:13,460 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:13,460 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:13,460 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:13,460 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:13,460 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:13,461 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:13,462 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:13,463 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:13,463 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:13,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:13,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:13,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:13,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:13,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:13,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:13,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:13,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:13,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:13,464 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:13,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:13,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:13,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,466 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:13,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:13,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,470 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,470 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:13,470 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:13,470 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,470 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:13,470 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:13,470 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:13,470 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:13,470 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:13,470 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:13,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:13,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:13,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:13,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:13,471 DEBUG [Thread-4140] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:13,471 DEBUG [Thread-4140] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:13,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:13,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:13,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on 313b3df11d64a9396698e7102a97394f 2023-03-31 16:56:13,508 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 313b3df11d64a9396698e7102a97394f 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-31 16:56:13,508 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=A 2023-03-31 16:56:13,508 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:56:13,508 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=B 2023-03-31 16:56:13,508 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:56:13,508 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=C 2023-03-31 16:56:13,508 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:56:13,521 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303313cc852c6456f4e8794949816c9ca3f5a to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303313cc852c6456f4e8794949816c9ca3f5a 2023-03-31 16:56:13,526 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=97, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/A/6d962cc812da4b0181cc0215dff4d31b 2023-03-31 16:56:13,541 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=97 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/B/d249c4bb5e044bb2abaade955913e63d 2023-03-31 16:56:13,556 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:13,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49568 deadline: 1680281833553, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,556 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:13,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49566 deadline: 1680281833554, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,571 DEBUG [Thread-4140] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:13,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:13,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:13,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:13,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:13,586 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:13,586 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:13,586 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:13,586 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:13,587 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:13,587 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:13,588 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,588 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,588 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:13,592 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:13,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:13,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:13,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:13,593 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:13,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:13,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:13,594 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:13,600 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:13,600 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:13,600 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:13,600 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:13,600 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=97 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/C/ca066de47a0d41f182a2ba3e703caa04 2023-03-31 16:56:13,600 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:13,605 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/A/6d962cc812da4b0181cc0215dff4d31b as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/6d962cc812da4b0181cc0215dff4d31b 2023-03-31 16:56:13,608 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:13,608 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:13,608 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:13,608 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. started... 2023-03-31 16:56:13,608 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. as already flushing 2023-03-31 16:56:13,608 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:13,608 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:13,608 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:13,608 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,608 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:13,609 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:13,610 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/6d962cc812da4b0181cc0215dff4d31b, entries=200, sequenceid=97, filesize=31.9 K 2023-03-31 16:56:13,610 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,610 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:13,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:13,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:13,611 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/B/d249c4bb5e044bb2abaade955913e63d as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/d249c4bb5e044bb2abaade955913e63d 2023-03-31 16:56:13,611 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:13,611 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:13,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:13,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:13,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:13,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:13,613 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:13,613 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@494e0557[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:13,613 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:13,613 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:13,614 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:13,614 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,614 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:13,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:13,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:13,616 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:13,616 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:13,616 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:13,616 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/d249c4bb5e044bb2abaade955913e63d, entries=150, sequenceid=97, filesize=11.6 K 2023-03-31 16:56:13,617 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/C/ca066de47a0d41f182a2ba3e703caa04 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/ca066de47a0d41f182a2ba3e703caa04 2023-03-31 16:56:13,618 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:13,618 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:13,618 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,619 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:13,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:13,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:13,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:13,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:13,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:13,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:13,621 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/ca066de47a0d41f182a2ba3e703caa04, entries=150, sequenceid=97, filesize=11.6 K 2023-03-31 16:56:13,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:13,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:13,622 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 313b3df11d64a9396698e7102a97394f in 114ms, sequenceid=97, compaction requested=true 2023-03-31 16:56:13,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:13,622 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:13,622 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:13,622 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:13,622 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 313b3df11d64a9396698e7102a97394f: 2023-03-31 16:56:13,622 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:13,622 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:13,623 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:56:13,623 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:56:13,623 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-31 16:56:13,623 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 3 compacting, 2 eligible, 16 blocking 2023-03-31 16:56:13,623 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-31 16:56:13,623 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-31 16:56:13,623 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. because compaction request was cancelled 2023-03-31 16:56:13,623 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 16:56:13,624 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:13,624 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:13,624 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:13,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:13,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:13,624 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 16:56:13,625 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): 313b3df11d64a9396698e7102a97394f/B is initiating minor compaction (all files) 2023-03-31 16:56:13,625 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 313b3df11d64a9396698e7102a97394f/B in TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:13,625 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/58787737611d4d6fa2826a94387010f7, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/db0e267fa452488fae44c86b816db169, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/d249c4bb5e044bb2abaade955913e63d] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp, totalSize=34.9 K 2023-03-31 16:56:13,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:13,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:13,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:13,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:13,625 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 58787737611d4d6fa2826a94387010f7, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1680281772728 2023-03-31 16:56:13,625 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:13,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:13,626 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting db0e267fa452488fae44c86b816db169, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1680281772771 2023-03-31 16:56:13,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:13,626 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting d249c4bb5e044bb2abaade955913e63d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1680281773393 2023-03-31 16:56:13,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:13,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:13,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:13,627 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:13,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:13,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,633 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:13,633 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:13,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:13,633 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:13,634 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:13,634 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:13,634 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:13,634 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:13,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:13,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:13,634 DEBUG [Thread-4140] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:13,634 DEBUG [Thread-4140] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:13,634 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:13,634 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,634 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:13,634 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:13,634 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:13,634 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:56:13,636 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): 313b3df11d64a9396698e7102a97394f#B#compaction#578 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:56:13,657 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/B/1df3c8a1e3a44325927604d51467e310 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/1df3c8a1e3a44325927604d51467e310 2023-03-31 16:56:13,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on 313b3df11d64a9396698e7102a97394f 2023-03-31 16:56:13,660 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 313b3df11d64a9396698e7102a97394f 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-03-31 16:56:13,660 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=A 2023-03-31 16:56:13,661 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:56:13,661 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=B 2023-03-31 16:56:13,661 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:56:13,661 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=C 2023-03-31 16:56:13,661 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:56:13,667 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 313b3df11d64a9396698e7102a97394f/B of 313b3df11d64a9396698e7102a97394f into 1df3c8a1e3a44325927604d51467e310(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:56:13,667 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 313b3df11d64a9396698e7102a97394f: 2023-03-31 16:56:13,667 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f., storeName=313b3df11d64a9396698e7102a97394f/B, priority=13, startTime=1680281773623; duration=0sec 2023-03-31 16:56:13,667 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:56:13,667 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 16:56:13,668 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 16:56:13,668 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): 313b3df11d64a9396698e7102a97394f/C is initiating minor compaction (all files) 2023-03-31 16:56:13,668 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 313b3df11d64a9396698e7102a97394f/C in TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:13,668 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/dd27abe1f1c942da90188f1f69ba5ee2, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/5f9c96f314314f298418c829edf13340, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/ca066de47a0d41f182a2ba3e703caa04] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp, totalSize=34.9 K 2023-03-31 16:56:13,668 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting dd27abe1f1c942da90188f1f69ba5ee2, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1680281772728 2023-03-31 16:56:13,669 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 5f9c96f314314f298418c829edf13340, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1680281772771 2023-03-31 16:56:13,669 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting ca066de47a0d41f182a2ba3e703caa04, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1680281773393 2023-03-31 16:56:13,674 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): 313b3df11d64a9396698e7102a97394f#C#compaction#580 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:56:13,681 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230331883da464badd4f29ad3ef30f998a720d to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331883da464badd4f29ad3ef30f998a720d 2023-03-31 16:56:13,686 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:13,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49566 deadline: 1680281833683, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,686 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:13,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49568 deadline: 1680281833684, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,692 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=118, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/A/5e649225813a458d8a1636868b48bb1b 2023-03-31 16:56:13,710 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=118 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/B/f244f7e767144fcabbc899e4573c4ced 2023-03-31 16:56:13,719 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=118 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/C/60babfe39e684c5c9af1b356d6a6b69d 2023-03-31 16:56:13,722 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/A/5e649225813a458d8a1636868b48bb1b as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/5e649225813a458d8a1636868b48bb1b 2023-03-31 16:56:13,726 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/5e649225813a458d8a1636868b48bb1b, entries=150, sequenceid=118, filesize=25.1 K 2023-03-31 16:56:13,727 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/B/f244f7e767144fcabbc899e4573c4ced as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/f244f7e767144fcabbc899e4573c4ced 2023-03-31 16:56:13,730 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/A/7c728df067244d8f86ca8fb8483f4c3d as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/7c728df067244d8f86ca8fb8483f4c3d 2023-03-31 16:56:13,731 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/f244f7e767144fcabbc899e4573c4ced, entries=150, sequenceid=118, filesize=11.6 K 2023-03-31 16:56:13,732 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/C/60babfe39e684c5c9af1b356d6a6b69d as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/60babfe39e684c5c9af1b356d6a6b69d 2023-03-31 16:56:13,734 DEBUG [Thread-4140] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:13,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:13,736 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 313b3df11d64a9396698e7102a97394f/A of 313b3df11d64a9396698e7102a97394f into 7c728df067244d8f86ca8fb8483f4c3d(size=25.2 K), total size for store is 107.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:56:13,736 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 313b3df11d64a9396698e7102a97394f: 2023-03-31 16:56:13,736 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f., storeName=313b3df11d64a9396698e7102a97394f/A, priority=13, startTime=1680281773218; duration=0sec 2023-03-31 16:56:13,736 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:56:13,736 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/60babfe39e684c5c9af1b356d6a6b69d, entries=150, sequenceid=118, filesize=11.6 K 2023-03-31 16:56:13,737 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=53.67 KB/54960 for 313b3df11d64a9396698e7102a97394f in 77ms, sequenceid=118, compaction requested=true 2023-03-31 16:56:13,737 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 313b3df11d64a9396698e7102a97394f: 2023-03-31 16:56:13,737 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:56:13,737 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 16:56:13,737 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:56:13,737 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:13,737 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-31 16:56:13,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:13,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:13,740 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:13,740 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:13,740 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:13,740 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:13,740 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:13,740 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:13,744 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 110027 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 16:56:13,744 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1912): 313b3df11d64a9396698e7102a97394f/A is initiating minor compaction (all files) 2023-03-31 16:56:13,744 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 313b3df11d64a9396698e7102a97394f/A in TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:13,744 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/7c728df067244d8f86ca8fb8483f4c3d, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/36e4511ef13f47fcb823cbd9645b2920, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/6d962cc812da4b0181cc0215dff4d31b, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/5e649225813a458d8a1636868b48bb1b] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp, totalSize=107.4 K 2023-03-31 16:56:13,745 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 7c728df067244d8f86ca8fb8483f4c3d, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1680281772728 2023-03-31 16:56:13,745 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,745 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:13,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:13,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:13,745 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,745 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:13,745 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 36e4511ef13f47fcb823cbd9645b2920, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1680281772771 2023-03-31 16:56:13,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:13,746 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:13,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:13,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:13,746 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:13,746 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 6d962cc812da4b0181cc0215dff4d31b, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1680281773393 2023-03-31 16:56:13,746 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 5e649225813a458d8a1636868b48bb1b, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=118, earliestPutTs=1680281773544 2023-03-31 16:56:13,752 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:13,752 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:13,752 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:13,752 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:13,752 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:13,756 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:13,756 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:13,756 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:13,756 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. started... 2023-03-31 16:56:13,756 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 313b3df11d64a9396698e7102a97394f 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-31 16:56:13,756 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=A 2023-03-31 16:56:13,756 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:56:13,756 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=B 2023-03-31 16:56:13,757 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:56:13,757 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=C 2023-03-31 16:56:13,757 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:56:13,775 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] throttle.PressureAwareThroughputController(145): 313b3df11d64a9396698e7102a97394f#A#compaction#583 average throughput is 0.86 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:56:13,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on 313b3df11d64a9396698e7102a97394f 2023-03-31 16:56:13,789 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. as already flushing 2023-03-31 16:56:13,802 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023033195b93d84fe6740f89676408aee92e41f to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033195b93d84fe6740f89676408aee92e41f 2023-03-31 16:56:13,824 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680281773812, max timestamp=1680281773822Num uniques 859; 2023-03-31 16:56:13,824 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680281773812, max timestamp=1680281773822Num uniques 859; 2023-03-31 16:56:13,824 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 16:56:13,824 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 16:56:13,837 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=131, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/A/892b82e4bcde44d9a096cbaa9a7a846c 2023-03-31 16:56:13,849 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=131 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/B/beaa85c63e094a9dbf109dd71b3a8d21 2023-03-31 16:56:13,858 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=131 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/C/6694aed74b3949ed872eba616ce0665c 2023-03-31 16:56:13,862 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/A/892b82e4bcde44d9a096cbaa9a7a846c as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/892b82e4bcde44d9a096cbaa9a7a846c 2023-03-31 16:56:13,875 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/892b82e4bcde44d9a096cbaa9a7a846c, entries=150, sequenceid=131, filesize=25.2 K 2023-03-31 16:56:13,875 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/B/beaa85c63e094a9dbf109dd71b3a8d21 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/beaa85c63e094a9dbf109dd71b3a8d21 2023-03-31 16:56:13,880 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/beaa85c63e094a9dbf109dd71b3a8d21, entries=150, sequenceid=131, filesize=11.7 K 2023-03-31 16:56:13,880 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/C/6694aed74b3949ed872eba616ce0665c as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/6694aed74b3949ed872eba616ce0665c 2023-03-31 16:56:13,883 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:13,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49566 deadline: 1680281833878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,884 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:13,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49568 deadline: 1680281833878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,884 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/6694aed74b3949ed872eba616ce0665c, entries=150, sequenceid=131, filesize=11.7 K 2023-03-31 16:56:13,885 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=123.00 KB/125950 for 313b3df11d64a9396698e7102a97394f in 129ms, sequenceid=131, compaction requested=true 2023-03-31 16:56:13,885 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 313b3df11d64a9396698e7102a97394f: 2023-03-31 16:56:13,885 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:13,885 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:13,885 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:13,885 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,890 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:13,890 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:13,891 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,891 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:13,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:13,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:13,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:13,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:13,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:13,893 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:13,893 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:13,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:13,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:13,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@157dfd53[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:13,893 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:13,894 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:13,895 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:13,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:13,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:13,895 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:13,895 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:13,895 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:13,896 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,896 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:13,897 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:13,897 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:13,897 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:13,897 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:13,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:13,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:13,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:13,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:13,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:13,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:13,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:13,900 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:13,900 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:13,900 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:13,900 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:13,900 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:13,900 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:13,902 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-31 16:56:13,902 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:13,902 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:13,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:13,902 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:13,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:13,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:13,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:13,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:13,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:13,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:13,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:13,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:13,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:13,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:13,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:13,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:13,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:13,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,906 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on 313b3df11d64a9396698e7102a97394f 2023-03-31 16:56:13,907 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 313b3df11d64a9396698e7102a97394f 3/3 column families, dataSize=129.71 KB heapSize=373.55 KB 2023-03-31 16:56:13,908 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=A 2023-03-31 16:56:13,908 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:56:13,908 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=B 2023-03-31 16:56:13,908 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:56:13,908 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=C 2023-03-31 16:56:13,908 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:56:13,908 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,910 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:13,910 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:13,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:13,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:13,911 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,911 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:13,911 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:13,911 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,911 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:13,911 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:13,911 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:13,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:13,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:13,912 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:13,912 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:13,912 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:13,913 DEBUG [Thread-4140] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:13,913 DEBUG [Thread-4140] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:13,923 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:13,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49608 deadline: 1680281833921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,928 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:13,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49602 deadline: 1680281833922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,928 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:13,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49618 deadline: 1680281833923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,962 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230331a7896db82e8d4e71ade75f0f1de5e877 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331a7896db82e8d4e71ade75f0f1de5e877 2023-03-31 16:56:13,989 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:13,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49566 deadline: 1680281833985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:13,989 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:13,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49568 deadline: 1680281833985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,013 DEBUG [Thread-4140] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:14,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:14,015 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:14,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:14,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:14,017 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:14,017 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:14,017 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:14,017 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:14,018 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:14,018 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:14,019 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:14,019 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:14,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:14,019 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,019 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:14,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:14,020 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:14,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:14,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:14,021 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:14,027 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:14,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49608 deadline: 1680281834025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,028 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:14,028 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:14,028 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:14,028 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:14,028 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:14,028 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:14,028 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:14,028 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:14,028 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. started... 2023-03-31 16:56:14,028 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. as already flushing 2023-03-31 16:56:14,028 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:14,029 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:14,029 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:14,029 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,029 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:14,029 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:14,030 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:14,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:14,031 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:14,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:14,031 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:14,031 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:14,031 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:14,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:14,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49618 deadline: 1680281834030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:14,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:14,033 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:14,033 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b93312d[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:14,033 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:14,033 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:14,034 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:14,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49602 deadline: 1680281834031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,035 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:14,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:14,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:14,035 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:14,035 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:14,035 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:14,036 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,036 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:14,038 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,038 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:14,038 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:14,038 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:14,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:14,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:14,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:14,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:14,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:14,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:14,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:14,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:14,042 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:14,042 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:14,042 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:14,042 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:14,042 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:14,042 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:14,044 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:14,044 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:14,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:14,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:14,044 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:14,044 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:14,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:14,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:14,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:14,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:14,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:14,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:14,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:14,046 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:14,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:14,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:14,049 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,049 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:14,049 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:14,049 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,049 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:14,049 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:14,049 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:14,049 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:14,049 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:14,049 DEBUG [Thread-4140] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:14,049 DEBUG [Thread-4140] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:14,049 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:14,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:14,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:14,049 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:14,049 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:14,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:14,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:14,050 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:56:14,098 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/C/029814b4ca664c32bc1461d3a3d075ca as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/029814b4ca664c32bc1461d3a3d075ca 2023-03-31 16:56:14,101 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 313b3df11d64a9396698e7102a97394f/C of 313b3df11d64a9396698e7102a97394f into 029814b4ca664c32bc1461d3a3d075ca(size=11.8 K), total size for store is 35.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:56:14,102 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 313b3df11d64a9396698e7102a97394f: 2023-03-31 16:56:14,102 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f., storeName=313b3df11d64a9396698e7102a97394f/C, priority=13, startTime=1680281773623; duration=0sec 2023-03-31 16:56:14,102 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-31 16:56:14,102 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 16:56:14,102 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35914 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 16:56:14,103 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): 313b3df11d64a9396698e7102a97394f/C is initiating minor compaction (all files) 2023-03-31 16:56:14,103 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 313b3df11d64a9396698e7102a97394f/C in TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:14,103 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/029814b4ca664c32bc1461d3a3d075ca, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/60babfe39e684c5c9af1b356d6a6b69d, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/6694aed74b3949ed872eba616ce0665c] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp, totalSize=35.1 K 2023-03-31 16:56:14,103 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 029814b4ca664c32bc1461d3a3d075ca, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1680281773393 2023-03-31 16:56:14,103 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 60babfe39e684c5c9af1b356d6a6b69d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=118, earliestPutTs=1680281773544 2023-03-31 16:56:14,104 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 6694aed74b3949ed872eba616ce0665c, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=131, earliestPutTs=1680281773662 2023-03-31 16:56:14,124 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): 313b3df11d64a9396698e7102a97394f#C#compaction#588 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:56:14,149 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/C/c9844faef41c4de88e6af9777b711d5b as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/c9844faef41c4de88e6af9777b711d5b 2023-03-31 16:56:14,149 DEBUG [Thread-4140] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:14,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:14,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:14,156 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 313b3df11d64a9396698e7102a97394f/C of 313b3df11d64a9396698e7102a97394f into c9844faef41c4de88e6af9777b711d5b(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:56:14,156 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 313b3df11d64a9396698e7102a97394f: 2023-03-31 16:56:14,156 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f., storeName=313b3df11d64a9396698e7102a97394f/C, priority=13, startTime=1680281773737; duration=0sec 2023-03-31 16:56:14,156 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:56:14,156 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 16:56:14,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:14,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:14,160 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:14,160 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:14,160 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:14,160 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35914 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 16:56:14,160 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:14,160 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): 313b3df11d64a9396698e7102a97394f/B is initiating minor compaction (all files) 2023-03-31 16:56:14,160 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 313b3df11d64a9396698e7102a97394f/B in TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:14,160 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/1df3c8a1e3a44325927604d51467e310, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/f244f7e767144fcabbc899e4573c4ced, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/beaa85c63e094a9dbf109dd71b3a8d21] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp, totalSize=35.1 K 2023-03-31 16:56:14,160 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:14,161 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:14,161 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 1df3c8a1e3a44325927604d51467e310, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1680281773393 2023-03-31 16:56:14,161 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting f244f7e767144fcabbc899e4573c4ced, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=118, earliestPutTs=1680281773544 2023-03-31 16:56:14,161 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting beaa85c63e094a9dbf109dd71b3a8d21, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=131, earliestPutTs=1680281773662 2023-03-31 16:56:14,163 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,163 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:14,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:14,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:14,163 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,163 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:14,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:14,164 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:14,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:14,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:14,165 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:14,165 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:14,165 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:14,165 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:14,165 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:14,165 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:14,165 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:14,165 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:14,166 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. started... 2023-03-31 16:56:14,166 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. as already flushing 2023-03-31 16:56:14,166 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:14,166 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:14,166 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:14,166 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,166 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:14,166 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:14,168 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:14,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:14,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:14,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:14,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:14,169 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): 313b3df11d64a9396698e7102a97394f#B#compaction#589 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:56:14,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:14,170 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:14,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:14,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f101f99[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:14,170 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:14,170 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:14,170 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:14,170 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:14,171 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:14,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:14,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:14,171 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:14,171 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:14,171 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,172 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:14,171 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:14,173 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:14,173 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:14,173 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:14,173 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:14,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:14,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:14,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:14,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:14,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:14,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:14,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:14,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:14,176 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:14,176 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:14,176 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:14,176 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:14,176 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:14,178 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:14,178 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:14,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:14,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:14,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:14,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:14,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:14,178 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:14,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:14,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:14,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:14,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:14,181 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:14,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:14,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:14,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,182 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:14,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:14,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,185 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:14,185 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:14,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:14,185 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:14,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:14,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:14,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:14,185 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,186 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:14,186 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:14,186 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,186 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:14,186 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:14,186 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:14,185 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:14,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:14,186 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:56:14,186 DEBUG [Thread-4140] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:14,186 DEBUG [Thread-4140] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:14,193 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:14,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49568 deadline: 1680281834191, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,195 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:14,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49566 deadline: 1680281834191, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,207 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/B/77318e9a09de4e2d8c7cbc694418ceb4 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/77318e9a09de4e2d8c7cbc694418ceb4 2023-03-31 16:56:14,212 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 313b3df11d64a9396698e7102a97394f/B of 313b3df11d64a9396698e7102a97394f into 77318e9a09de4e2d8c7cbc694418ceb4(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:56:14,212 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 313b3df11d64a9396698e7102a97394f: 2023-03-31 16:56:14,212 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f., storeName=313b3df11d64a9396698e7102a97394f/B, priority=13, startTime=1680281773737; duration=0sec 2023-03-31 16:56:14,213 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:56:14,232 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:14,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49608 deadline: 1680281834229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,237 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:14,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49602 deadline: 1680281834235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,238 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:14,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49618 deadline: 1680281834236, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,249 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/A/41fc4ecadcd24322b94db4199aff434c as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/41fc4ecadcd24322b94db4199aff434c 2023-03-31 16:56:14,254 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 313b3df11d64a9396698e7102a97394f/A of 313b3df11d64a9396698e7102a97394f into 41fc4ecadcd24322b94db4199aff434c(size=25.4 K), total size for store is 50.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:56:14,254 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 313b3df11d64a9396698e7102a97394f: 2023-03-31 16:56:14,255 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f., storeName=313b3df11d64a9396698e7102a97394f/A, priority=12, startTime=1680281773737; duration=0sec 2023-03-31 16:56:14,255 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:56:14,286 DEBUG [Thread-4140] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:14,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:14,288 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:14,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:14,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:14,290 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:14,290 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:14,290 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:14,290 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:14,290 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:14,290 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:14,292 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,292 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:14,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:14,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:14,292 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,292 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:14,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:14,293 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:14,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:14,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:14,294 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:14,294 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:14,294 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:14,294 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:14,295 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:14,295 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:14,295 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:14,295 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:14,295 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:14,295 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. started... 2023-03-31 16:56:14,295 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. as already flushing 2023-03-31 16:56:14,295 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:14,295 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:14,295 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:14,295 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,295 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:14,295 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:14,301 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,301 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:14,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:14,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:14,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:14,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:14,302 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:14,302 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:14,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:14,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:14,303 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:14,303 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66cddd91[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:14,303 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:14,303 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:14,304 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,304 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:14,304 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:14,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:14,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:14,305 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:14,305 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:14,305 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:14,307 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:14,307 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:14,307 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,307 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:14,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:14,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:14,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:14,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:14,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:14,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:14,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:14,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:14,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:14,310 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:14,310 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:14,310 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:14,310 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:14,310 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:14,312 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:14,312 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:14,312 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:14,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:14,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:14,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:14,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:14,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:14,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:14,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:14,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:14,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:14,313 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:14,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:14,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:14,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,314 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:14,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:14,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,317 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,317 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:14,317 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:14,318 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,318 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:14,318 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:14,318 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:14,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:14,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:14,318 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:14,318 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:14,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:14,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:14,318 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:14,318 DEBUG [Thread-4140] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:14,318 DEBUG [Thread-4140] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:14,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:14,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:14,381 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=163, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/A/60d55b632371423ab562f8df426556d9 2023-03-31 16:56:14,418 DEBUG [Thread-4140] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:14,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:14,420 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:14,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:14,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:14,422 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:14,422 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:14,422 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:14,422 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:14,422 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:14,422 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:14,423 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:14,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:14,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:14,424 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:14,424 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,424 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:14,424 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:14,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:14,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:14,425 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:14,425 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:14,425 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:14,425 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:14,425 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:14,425 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:14,425 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:14,426 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:14,426 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:14,426 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. started... 2023-03-31 16:56:14,426 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. as already flushing 2023-03-31 16:56:14,426 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:14,426 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:14,426 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:14,426 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,426 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:14,426 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:14,427 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,427 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:14,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:14,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:14,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:14,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:14,428 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:14,429 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:14,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:14,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:14,430 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:14,430 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c20237e[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:14,430 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:14,430 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:14,431 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:14,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:14,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:14,431 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:14,431 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:14,431 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:14,432 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,432 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:14,435 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:14,435 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:14,435 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:14,435 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:14,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:14,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:14,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:14,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:14,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:14,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:14,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:14,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:14,438 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:14,438 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:14,438 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:14,438 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:14,438 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:14,439 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:14,440 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:14,440 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:14,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:14,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:14,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:14,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:14,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:14,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:14,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:14,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:14,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:14,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:14,440 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:14,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:14,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:14,442 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:14,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,445 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:14,445 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:14,445 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:14,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:14,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:14,445 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:14,445 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,445 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:14,445 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:14,445 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:14,445 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:14,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:14,445 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:14,445 DEBUG [Thread-4140] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:14,445 DEBUG [Thread-4140] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:14,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:14,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:14,497 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:14,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49568 deadline: 1680281834494, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,500 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:14,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49566 deadline: 1680281834498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,536 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:14,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49608 deadline: 1680281834535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,541 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:14,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49602 deadline: 1680281834540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,543 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:14,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49618 deadline: 1680281834541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,546 DEBUG [Thread-4140] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:14,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:14,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:14,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:14,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:14,549 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:14,549 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:14,550 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:14,550 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:14,551 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:14,551 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:14,553 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,553 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:14,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:14,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:14,553 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,553 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:14,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:14,554 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:14,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:14,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:14,554 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:14,554 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:14,554 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:14,554 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:14,554 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:14,555 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:14,556 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:14,556 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:14,556 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:14,556 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. started... 2023-03-31 16:56:14,556 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. as already flushing 2023-03-31 16:56:14,556 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:14,556 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:14,556 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:14,556 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,556 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:14,556 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:14,559 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,559 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:14,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:14,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:14,559 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:14,559 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:14,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:14,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:14,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:14,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:14,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:14,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e88638e[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:14,561 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:14,561 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:14,562 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:14,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:14,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:14,563 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:14,563 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:14,563 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:14,563 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,563 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:14,564 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:14,564 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:14,564 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,564 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:14,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:14,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:14,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:14,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:14,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:14,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:14,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:14,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:14,568 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:14,568 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:14,568 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:14,568 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:14,568 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:14,568 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:14,570 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:14,570 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:14,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:14,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:14,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:14,570 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:14,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:14,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:14,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:14,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:14,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:14,571 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:14,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:14,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:14,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:14,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,573 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:14,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:14,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,578 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,578 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:14,578 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:14,578 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:14,578 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:56:14,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:14,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:14,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:14,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:14,578 DEBUG [Thread-4140] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:14,578 DEBUG [Thread-4140] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:14,578 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:14,578 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:14,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:14,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:14,578 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,579 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:14,579 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:14,579 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:14,678 DEBUG [Thread-4140] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:14,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:14,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:14,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:14,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:14,682 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:14,682 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:14,682 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:14,682 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:14,682 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:14,683 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:14,684 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:14,684 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:14,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:14,684 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:14,684 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:14,685 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:14,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:14,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:14,685 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:14,685 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:14,685 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:14,685 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:14,685 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:14,685 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:14,686 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:14,686 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:14,686 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:14,686 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. started... 2023-03-31 16:56:14,686 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. as already flushing 2023-03-31 16:56:14,686 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:14,686 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:14,686 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:14,686 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,686 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:14,686 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:14,687 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,687 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:14,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:14,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:14,688 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:14,688 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:14,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:14,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:14,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:14,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:14,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:14,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@713b4d11[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:14,689 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:14,689 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:14,694 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:14,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:14,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:14,694 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:14,694 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:14,694 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:14,694 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,694 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:14,696 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,696 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:14,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:14,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:14,696 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:14,696 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:14,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:14,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:14,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:14,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:14,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:14,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:14,698 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:14,698 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:14,698 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:14,698 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:14,698 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:14,698 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:14,700 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:14,700 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:14,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:14,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:14,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:14,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:14,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:14,700 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:14,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:14,701 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:14,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:14,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:14,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:14,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:14,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:14,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,702 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:14,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:14,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,704 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,704 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:14,704 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:14,704 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:14,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:14,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:14,704 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,705 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:14,705 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:14,705 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:14,704 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:14,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:14,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:14,705 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:14,705 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:14,705 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:14,706 DEBUG [Thread-4140] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:14,706 DEBUG [Thread-4140] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:14,793 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=64.85 KB at sequenceid=163 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/B/b253ed4b4dc84085a4474218345fd11d 2023-03-31 16:56:14,806 DEBUG [Thread-4140] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:14,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:14,808 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:14,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:14,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:14,809 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:14,809 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:14,809 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:14,809 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:14,810 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:14,810 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:14,812 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:14,812 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:14,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:14,812 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,812 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:14,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:14,813 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:14,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:14,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:14,813 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:14,813 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:14,813 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:14,813 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:14,813 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:14,813 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:14,814 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:14,814 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:14,814 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:14,814 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. started... 2023-03-31 16:56:14,814 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. as already flushing 2023-03-31 16:56:14,814 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:14,814 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:14,814 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:14,814 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,814 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:14,814 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:14,815 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:14,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:14,816 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:14,816 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:14,816 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:14,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:14,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:14,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:14,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:14,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:14,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@791227f2[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:14,818 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:14,818 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:14,819 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:14,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:14,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:14,819 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:14,820 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:14,820 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,820 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:14,820 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:14,821 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:14,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:14,822 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:14,822 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:14,822 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:14,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:14,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:14,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:14,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:14,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:14,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:14,824 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:14,824 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:14,824 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:14,824 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:14,824 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:14,824 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:14,827 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:14,827 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:14,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:14,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:14,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:14,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:14,827 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:14,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:14,828 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:14,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:14,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:14,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:14,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:14,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:14,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:14,829 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:14,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:14,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,832 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,832 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:14,832 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:14,832 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:56:14,832 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:14,832 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:14,832 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:14,832 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:14,832 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:14,832 DEBUG [Thread-4140] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:14,832 DEBUG [Thread-4140] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:14,832 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:14,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:14,832 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:14,832 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:14,832 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:14,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:14,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:14,932 DEBUG [Thread-4140] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:14,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:14,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:14,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:14,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:14,936 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:14,936 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:14,936 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:14,936 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:14,936 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:14,936 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:14,938 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:14,938 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:14,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:14,938 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,938 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:14,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:14,940 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:14,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:14,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:14,940 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:14,940 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:14,941 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:14,941 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:14,941 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:14,941 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:14,941 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:14,941 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:14,941 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:14,941 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. started... 2023-03-31 16:56:14,941 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. as already flushing 2023-03-31 16:56:14,941 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:14,941 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:14,941 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:14,941 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,941 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:14,941 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:14,943 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,943 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:14,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:14,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:14,943 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:14,943 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:14,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:14,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:14,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:14,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:14,945 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:14,945 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6434d07d[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:14,945 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:14,945 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:14,947 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:14,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:14,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:14,948 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:14,948 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:14,948 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:14,948 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,948 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:14,949 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:14,949 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:14,949 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,949 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:14,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:14,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:14,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:14,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:14,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:14,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:14,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:14,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:14,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:14,955 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:14,955 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:14,955 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:14,955 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:14,955 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:14,957 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:14,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:14,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:14,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:14,957 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:14,957 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:14,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:14,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:14,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:14,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:14,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:14,957 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:14,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:14,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:14,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:14,959 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:14,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:14,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,961 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:14,961 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:14,961 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,961 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:14,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:14,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:14,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:14,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:14,961 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:56:14,961 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:14,961 DEBUG [Thread-4140] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:14,962 DEBUG [Thread-4140] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:14,961 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:14,961 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:14,962 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:14,962 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:14,962 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:14,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:14,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:15,001 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:15,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49568 deadline: 1680281834998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:15,009 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:15,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49566 deadline: 1680281835005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:15,040 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:15,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49608 deadline: 1680281835040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:15,044 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:15,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49602 deadline: 1680281835043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:15,048 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:15,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49618 deadline: 1680281835047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:15,062 DEBUG [Thread-4140] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:15,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:15,064 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:15,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:15,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:15,065 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:15,065 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:15,065 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:15,065 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:15,065 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:15,066 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:15,068 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:15,068 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:15,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:15,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:15,068 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:15,068 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:15,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:15,069 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:15,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:15,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:15,069 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:15,069 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:15,070 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:15,070 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:15,070 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:15,070 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:15,070 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:15,070 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:15,070 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:15,070 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. started... 2023-03-31 16:56:15,070 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. as already flushing 2023-03-31 16:56:15,070 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:15,070 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:15,070 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:15,070 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:15,070 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:15,070 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:15,072 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:15,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:15,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:15,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:15,072 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:15,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:15,072 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:15,072 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:15,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:15,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:15,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:15,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:15,074 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:15,074 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@94d4b66[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:15,074 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:15,074 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:15,075 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:15,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:15,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:15,075 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:15,075 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:15,075 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:15,075 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:15,075 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:15,080 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:15,080 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:15,080 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:15,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:15,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:15,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:15,080 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:15,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:15,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:15,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:15,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:15,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:15,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:15,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:15,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:15,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:15,083 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:15,083 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:15,083 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:15,083 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:15,083 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:15,084 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:15,084 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:15,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:15,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:15,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:15,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:15,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:15,084 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:15,085 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:15,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:15,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:15,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:15,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:15,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:15,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:15,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:15,087 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:15,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:15,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:15,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:15,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:15,090 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:15,090 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:15,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:15,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:15,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:15,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:15,090 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:15,090 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:15,090 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:15,090 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:15,090 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:15,090 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:15,091 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:15,091 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:15,091 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:15,091 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:15,091 DEBUG [Thread-4140] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:15,091 DEBUG [Thread-4140] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:15,191 DEBUG [Thread-4140] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:15,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:15,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:15,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:15,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:15,194 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:15,195 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:15,195 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:15,195 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:15,195 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:15,195 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:15,197 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:15,197 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:15,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:15,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:15,197 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:15,197 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:15,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:15,197 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:15,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:15,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:15,198 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:15,198 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:15,198 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:15,198 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:15,198 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:15,198 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:15,198 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:15,198 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:15,198 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:15,198 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. started... 2023-03-31 16:56:15,199 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. as already flushing 2023-03-31 16:56:15,199 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:15,199 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:15,199 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:15,199 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:15,199 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:15,199 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:15,202 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:15,202 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:15,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:15,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:15,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:15,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:15,203 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:15,203 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:15,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:15,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:15,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:15,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:15,204 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=163 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/C/02c00cbf15df4f7ab7fc69d129eef442 2023-03-31 16:56:15,204 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:15,204 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2235dc9b[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:15,204 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:15,204 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:15,206 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:15,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:15,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:15,206 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:15,206 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:15,206 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:15,206 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:15,206 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:15,208 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:15,208 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:15,208 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:15,208 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:15,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:15,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:15,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:15,208 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/A/60d55b632371423ab562f8df426556d9 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/60d55b632371423ab562f8df426556d9 2023-03-31 16:56:15,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:15,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:15,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:15,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:15,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:15,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:15,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:15,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:15,210 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:15,210 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:15,211 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:15,211 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:15,211 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:15,211 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:15,211 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/60d55b632371423ab562f8df426556d9, entries=250, sequenceid=163, filesize=38.9 K 2023-03-31 16:56:15,212 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/B/b253ed4b4dc84085a4474218345fd11d as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/b253ed4b4dc84085a4474218345fd11d 2023-03-31 16:56:15,212 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:15,212 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:15,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:15,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:15,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:15,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:15,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:15,212 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:15,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:15,213 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:15,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:15,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:15,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:15,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:15,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:15,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:15,215 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:15,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:15,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:15,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:15,216 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/b253ed4b4dc84085a4474218345fd11d, entries=150, sequenceid=163, filesize=11.8 K 2023-03-31 16:56:15,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:15,217 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/C/02c00cbf15df4f7ab7fc69d129eef442 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/02c00cbf15df4f7ab7fc69d129eef442 2023-03-31 16:56:15,220 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/02c00cbf15df4f7ab7fc69d129eef442, entries=150, sequenceid=163, filesize=11.8 K 2023-03-31 16:56:15,221 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~136.42 KB/139690, heapSize ~393.42 KB/402864, currentSize=40.25 KB/41220 for 313b3df11d64a9396698e7102a97394f in 1314ms, sequenceid=163, compaction requested=true 2023-03-31 16:56:15,221 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 313b3df11d64a9396698e7102a97394f: 2023-03-31 16:56:15,221 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:56:15,221 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:56:15,221 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-03-31 16:56:15,221 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 16:56:15,221 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:56:15,222 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:15,222 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:15,222 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:15,222 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:15,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:15,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:15,222 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:15,222 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:15,222 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:15,222 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:15,222 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:56:15,222 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 91662 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 16:56:15,222 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:15,222 DEBUG [Thread-4140] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:15,222 DEBUG [Thread-4140] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:15,222 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-31 16:56:15,222 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:15,223 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:15,223 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:15,222 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1912): 313b3df11d64a9396698e7102a97394f/A is initiating minor compaction (all files) 2023-03-31 16:56:15,223 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 313b3df11d64a9396698e7102a97394f/A in TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:15,223 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/41fc4ecadcd24322b94db4199aff434c, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/892b82e4bcde44d9a096cbaa9a7a846c, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/60d55b632371423ab562f8df426556d9] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp, totalSize=89.5 K 2023-03-31 16:56:15,223 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-31 16:56:15,223 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. because compaction request was cancelled 2023-03-31 16:56:15,223 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-03-31 16:56:15,223 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 41fc4ecadcd24322b94db4199aff434c, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=118, earliestPutTs=1680281773544 2023-03-31 16:56:15,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:15,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:15,224 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 892b82e4bcde44d9a096cbaa9a7a846c, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=131, earliestPutTs=1680281773662 2023-03-31 16:56:15,224 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-31 16:56:15,224 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-31 16:56:15,224 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 60d55b632371423ab562f8df426556d9, keycount=250, bloomtype=ROW, size=38.9 K, encoding=NONE, compression=NONE, seqNum=163, earliestPutTs=1680281773865 2023-03-31 16:56:15,224 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. because compaction request was cancelled 2023-03-31 16:56:15,232 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] throttle.PressureAwareThroughputController(145): 313b3df11d64a9396698e7102a97394f#A#compaction#592 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:56:15,323 DEBUG [Thread-4140] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:15,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:15,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:15,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:15,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:15,329 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:15,329 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:15,329 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:15,329 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:15,329 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:15,330 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:15,331 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:15,331 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:15,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:15,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:15,331 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:15,331 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:15,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:15,332 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:15,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:15,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:15,332 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:15,332 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:15,332 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:15,333 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:15,333 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:15,333 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:15,333 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:15,333 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:15,333 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:15,333 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. started... 2023-03-31 16:56:15,333 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 313b3df11d64a9396698e7102a97394f 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-03-31 16:56:15,333 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=A 2023-03-31 16:56:15,333 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:56:15,333 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=B 2023-03-31 16:56:15,333 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:56:15,334 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=C 2023-03-31 16:56:15,334 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:56:15,348 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230331dbfb39a2a71948e7bc824d45664bb5c9 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331dbfb39a2a71948e7bc824d45664bb5c9 2023-03-31 16:56:15,353 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=176, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/A/2132eedb9a33436ca187dcc7974e49b3 2023-03-31 16:56:15,766 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=176 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/B/a433bbefb54840d68735d3333436d8f0 2023-03-31 16:56:15,776 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=176 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/C/435cc93dd6194ef2bc78a80c47d8af90 2023-03-31 16:56:15,780 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/A/2132eedb9a33436ca187dcc7974e49b3 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/2132eedb9a33436ca187dcc7974e49b3 2023-03-31 16:56:15,785 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/2132eedb9a33436ca187dcc7974e49b3, entries=150, sequenceid=176, filesize=25.3 K 2023-03-31 16:56:15,785 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/B/a433bbefb54840d68735d3333436d8f0 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/a433bbefb54840d68735d3333436d8f0 2023-03-31 16:56:15,789 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/a433bbefb54840d68735d3333436d8f0, entries=150, sequenceid=176, filesize=11.8 K 2023-03-31 16:56:15,790 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/C/435cc93dd6194ef2bc78a80c47d8af90 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/435cc93dd6194ef2bc78a80c47d8af90 2023-03-31 16:56:15,793 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/435cc93dd6194ef2bc78a80c47d8af90, entries=150, sequenceid=176, filesize=11.8 K 2023-03-31 16:56:15,794 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=0 B/0 for 313b3df11d64a9396698e7102a97394f in 461ms, sequenceid=176, compaction requested=true 2023-03-31 16:56:15,794 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 313b3df11d64a9396698e7102a97394f: 2023-03-31 16:56:15,794 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:15,794 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:15,794 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:15,794 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:15,794 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:15,794 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:15,796 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:15,796 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:15,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:15,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:15,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:15,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:15,796 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:15,796 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:15,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:15,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:15,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:15,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:15,798 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:15,798 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a704298[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:15,798 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:15,798 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:15,799 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:15,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:15,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:15,800 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:15,800 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:15,800 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:15,800 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:15,800 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:15,802 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:15,802 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:15,802 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:15,802 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:15,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:15,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:15,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:15,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:15,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:15,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:15,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:15,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:15,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:15,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:15,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:15,805 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:15,805 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:15,805 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:15,805 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:15,805 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:15,805 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:15,807 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:15,807 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:15,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:15,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:15,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:15,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:15,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:15,807 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:15,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:15,808 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:15,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:15,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:15,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:15,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:15,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:15,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:15,809 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:15,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:15,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:15,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:15,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:15,811 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:15,811 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:15,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:15,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:15,811 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:15,811 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:15,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:15,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:15,811 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:15,812 DEBUG [Thread-4140] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:15,812 DEBUG [Thread-4140] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:15,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:15,811 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:56:15,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:15,811 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:15,812 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:15,812 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:15,812 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:15,812 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:15,916 DEBUG [Thread-4140] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:15,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:15,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:15,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:15,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:15,920 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:15,920 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:15,920 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:15,920 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:15,921 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:15,921 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:15,922 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:15,922 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:15,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:15,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:15,922 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:15,923 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:15,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:15,923 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:15,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:15,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:15,923 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:15,924 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:15,924 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:15,924 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:15,924 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:15,927 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:15,927 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:15,927 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:15,927 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. started... 2023-03-31 16:56:15,927 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 313b3df11d64a9396698e7102a97394f: 2023-03-31 16:56:15,927 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:15,928 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:15,928 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:15,928 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:15,928 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:15,928 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:15,931 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:15,931 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:15,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:15,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:15,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:15,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:15,931 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:15,931 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:15,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:15,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:15,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:15,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:15,933 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:15,933 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@885c493[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:15,933 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:15,933 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:15,934 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:15,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:15,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:15,934 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:15,934 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:15,934 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:15,935 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:15,935 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:15,936 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:15,936 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:15,936 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:15,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:15,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:15,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:15,936 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:15,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:15,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:15,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:15,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:15,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:15,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:15,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:15,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:15,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:15,938 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:15,938 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:15,938 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:15,938 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:15,939 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:15,941 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:15,941 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:15,941 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:15,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:15,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:15,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:15,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:15,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:15,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:15,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:15,942 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:15,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:15,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:15,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:15,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:15,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:15,943 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:15,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:15,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:15,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:15,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:15,950 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:15,950 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:15,950 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:15,950 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:15,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:15,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:15,951 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:15,951 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:15,951 DEBUG [Thread-4140] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:15,951 DEBUG [Thread-4140] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:15,950 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:15,951 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:15,951 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:15,951 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:15,951 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:15,951 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:15,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:15,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:16,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on 313b3df11d64a9396698e7102a97394f 2023-03-31 16:56:16,044 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 313b3df11d64a9396698e7102a97394f 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-31 16:56:16,045 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=A 2023-03-31 16:56:16,045 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:56:16,045 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=B 2023-03-31 16:56:16,045 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:56:16,045 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=C 2023-03-31 16:56:16,045 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:56:16,051 DEBUG [Thread-4140] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:16,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:16,054 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:16,054 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/A/ee7a65ff8aac4e2baee210c3798c621c as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/ee7a65ff8aac4e2baee210c3798c621c 2023-03-31 16:56:16,059 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 313b3df11d64a9396698e7102a97394f/A of 313b3df11d64a9396698e7102a97394f into ee7a65ff8aac4e2baee210c3798c621c(size=25.6 K), total size for store is 50.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:56:16,059 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 313b3df11d64a9396698e7102a97394f: 2023-03-31 16:56:16,059 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f., storeName=313b3df11d64a9396698e7102a97394f/A, priority=13, startTime=1680281775221; duration=0sec 2023-03-31 16:56:16,059 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:56:16,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:16,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:16,059 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:16,059 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:16,059 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:16,059 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:16,060 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:16,060 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:16,064 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:16,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:16,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:16,065 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,065 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,065 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:16,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:16,065 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:16,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:16,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:16,066 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:16,066 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:16,066 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:16,066 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:16,066 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:16,067 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:16,068 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:16,068 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:16,068 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:16,068 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. started... 2023-03-31 16:56:16,068 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. as already flushing 2023-03-31 16:56:16,068 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303315c0714100447410fa4a23d82252a2a34 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303315c0714100447410fa4a23d82252a2a34 2023-03-31 16:56:16,068 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:16,068 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:16,068 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:16,068 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,068 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:16,068 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:16,071 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:16,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:16,071 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:16,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:16,072 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:16,072 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:16,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:16,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:16,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:16,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:16,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6bf0a89e[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:16,073 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:16,073 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:16,075 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,075 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:16,075 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1680281776062, max timestamp=1680281776073Num uniques 859; 2023-03-31 16:56:16,075 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1680281776062, max timestamp=1680281776073Num uniques 859; 2023-03-31 16:56:16,075 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 16:56:16,075 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 16:56:16,075 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:16,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:16,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:16,075 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:16,076 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:16,076 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:16,076 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680281776064, max timestamp=1680281776073Num uniques 859; 2023-03-31 16:56:16,076 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 16:56:16,078 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:16,078 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:16,078 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:16,078 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:16,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:16,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:16,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:16,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:16,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:16,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:16,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:16,082 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:16,082 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:16,082 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:16,082 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:16,082 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:16,082 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:16,084 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:16,084 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:16,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:16,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:16,084 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:16,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:16,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:16,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:16,085 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:16,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:16,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:16,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:16,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:16,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:16,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:16,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:16,087 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:16,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,092 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:16,092 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:16,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:16,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:16,092 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:16,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:16,093 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:16,093 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,093 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:16,093 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:16,093 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,093 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:16,093 DEBUG [Thread-4140] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:16,093 DEBUG [Thread-4140] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:16,093 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:16,093 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:16,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:16,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:16,095 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:16,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49608 deadline: 1680281836090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,096 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:16,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49602 deadline: 1680281836091, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,097 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:16,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49618 deadline: 1680281836094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,100 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:16,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49566 deadline: 1680281836096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,100 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:16,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49568 deadline: 1680281836096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,193 DEBUG [Thread-4140] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:16,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:16,195 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:16,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:16,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:16,196 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:16,196 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:16,196 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:16,196 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:16,197 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:16,197 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:16,198 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:16,198 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:16,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:16,198 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,198 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:16,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:16,199 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:16,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49608 deadline: 1680281836197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,199 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:16,199 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:16,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49602 deadline: 1680281836197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,199 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:16,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49618 deadline: 1680281836198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:16,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:16,199 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:16,200 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:16,200 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:16,200 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:16,200 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:16,200 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:16,200 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:16,200 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:16,200 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:16,200 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. started... 2023-03-31 16:56:16,200 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. as already flushing 2023-03-31 16:56:16,200 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:16,200 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:16,200 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:16,200 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,200 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:16,200 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:16,203 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:16,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49568 deadline: 1680281836201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,203 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:16,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49566 deadline: 1680281836201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,206 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:16,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:16,206 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:16,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:16,207 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:16,207 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:16,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:16,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:16,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:16,208 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:16,208 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67ad707f[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:16,208 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:16,208 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:16,210 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:16,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:16,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:16,210 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:16,210 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:16,210 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:16,210 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,210 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:16,211 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,211 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:16,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:16,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:16,211 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:16,212 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:16,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:16,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:16,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:16,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:16,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:16,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:16,214 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:16,214 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:16,214 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:16,214 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:16,214 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:16,214 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:16,216 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:16,216 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:16,216 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:16,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:16,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:16,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:16,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:16,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:16,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:16,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:16,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:16,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:16,217 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:16,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:16,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:16,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:16,218 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:16,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,221 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,221 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:16,221 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:16,221 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:16,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:16,222 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:16,221 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:16,222 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:16,222 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:16,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:16,222 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,222 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:16,222 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:16,222 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:16,222 DEBUG [Thread-4140] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:16,222 DEBUG [Thread-4140] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:16,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:16,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:16,322 DEBUG [Thread-4140] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:16,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:16,324 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:16,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:16,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:16,325 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:16,325 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:16,325 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:16,325 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:16,326 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:16,326 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:16,327 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:16,327 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:16,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:16,327 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:16,327 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:16,328 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:16,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:16,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:16,328 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:16,328 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:16,328 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:16,328 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:16,329 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:16,329 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:16,329 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:16,329 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:16,329 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:16,329 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. started... 2023-03-31 16:56:16,329 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. as already flushing 2023-03-31 16:56:16,329 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:16,329 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:16,329 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:16,329 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,329 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:16,329 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:16,331 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:16,331 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:16,331 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,331 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:16,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:16,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:16,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:16,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:16,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:16,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:16,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:16,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f018a90[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:16,333 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:16,333 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:16,334 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:16,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:16,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:16,335 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:16,335 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:16,335 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:16,335 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,335 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:16,339 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:16,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:16,339 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:16,339 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:16,339 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:16,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:16,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:16,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:16,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:16,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:16,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:16,341 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:16,341 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:16,341 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:16,341 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:16,341 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:16,341 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:16,343 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:16,343 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:16,343 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:16,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:16,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:16,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:16,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:16,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:16,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:16,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:16,344 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:16,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:16,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:16,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:16,345 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:16,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:16,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:16,347 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,347 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:16,347 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:16,347 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:16,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:16,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:16,347 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,347 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:16,348 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:16,348 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:16,347 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:16,348 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:56:16,347 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:16,347 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:16,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:16,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:16,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:16,348 DEBUG [Thread-4140] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:16,348 DEBUG [Thread-4140] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:16,403 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:16,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49608 deadline: 1680281836400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,403 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:16,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49602 deadline: 1680281836400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,403 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:16,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49618 deadline: 1680281836401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,408 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:16,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49568 deadline: 1680281836405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,408 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:16,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49566 deadline: 1680281836405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,448 DEBUG [Thread-4140] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:16,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:16,450 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:16,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:16,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:16,452 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:16,452 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:16,452 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:16,452 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:16,452 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:16,452 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:16,454 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:16,454 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:16,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:16,455 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,455 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:16,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:16,456 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:16,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:16,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:16,456 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:16,457 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:16,457 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:16,457 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:16,457 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:16,457 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:16,457 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:16,457 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:16,457 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:16,457 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. started... 2023-03-31 16:56:16,457 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. as already flushing 2023-03-31 16:56:16,457 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:16,457 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:16,457 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:16,457 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,457 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:16,457 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:16,460 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:16,460 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:16,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:16,460 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:16,460 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:16,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:16,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:16,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:16,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:16,462 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:16,462 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@98672bf[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:16,462 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:16,462 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:16,463 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:16,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:16,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:16,464 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:16,464 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:16,464 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:16,464 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,464 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:16,465 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,466 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:16,466 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:16,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:16,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:16,466 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:16,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:16,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:16,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:16,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:16,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:16,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:16,468 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:16,468 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:16,468 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:16,468 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:16,468 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:16,468 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:16,471 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:16,471 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:16,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:16,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:16,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:16,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:16,471 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:16,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:16,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:16,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:16,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:16,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:16,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:16,472 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:16,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:16,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,475 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:16,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:16,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,478 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,479 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:16,479 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:16,479 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:16,479 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,479 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:16,479 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:16,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:16,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:16,479 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:16,479 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:16,479 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:16,479 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:16,479 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:16,479 DEBUG [Thread-4140] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:16,479 DEBUG [Thread-4140] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:16,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:16,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:16,482 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=187, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/A/af62b9f582dd4018897c68167000682c 2023-03-31 16:56:16,579 DEBUG [Thread-4140] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:16,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:16,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:16,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:16,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:16,583 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:16,583 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:16,584 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:16,584 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:16,584 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:16,584 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:16,585 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,585 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:16,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:16,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:16,586 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,586 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:16,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:16,586 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:16,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:16,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:16,587 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:16,588 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:16,588 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:16,588 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:16,588 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:16,588 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:16,588 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:16,588 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:16,588 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:16,588 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. started... 2023-03-31 16:56:16,589 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. as already flushing 2023-03-31 16:56:16,589 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:16,589 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:16,589 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:16,589 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,589 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:16,589 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:16,590 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:16,590 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:16,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:16,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:16,591 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:16,591 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:16,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:16,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:16,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:16,592 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:16,592 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67dc9f49[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:16,592 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:16,592 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:16,595 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:16,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:16,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:16,595 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:16,595 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:16,595 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,595 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:16,595 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:16,597 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,597 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:16,597 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:16,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:16,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:16,597 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:16,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:16,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:16,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:16,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:16,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:16,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:16,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:16,601 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:16,601 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:16,601 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:16,601 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:16,601 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:16,603 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:16,603 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:16,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:16,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:16,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:16,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:16,603 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:16,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:16,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:16,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:16,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:16,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:16,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:16,604 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:16,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:16,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,606 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:16,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:16,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,609 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,609 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:16,609 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:16,609 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:16,609 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:16,609 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,609 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:16,609 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:16,609 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:16,609 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:16,609 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:16,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:16,609 DEBUG [Thread-4140] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:16,609 DEBUG [Thread-4140] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:16,609 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:16,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:16,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:16,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:16,708 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:16,708 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:16,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49608 deadline: 1680281836705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49602 deadline: 1680281836705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,708 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:16,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49618 deadline: 1680281836706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,710 DEBUG [Thread-4140] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:16,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:16,711 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:16,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:16,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:16,712 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:16,712 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:16,712 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:16,712 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:16,713 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:16,713 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:16,713 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:16,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49568 deadline: 1680281836709, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,713 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:16,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49566 deadline: 1680281836709, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,715 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,715 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:16,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:16,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:16,715 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,715 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:16,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:16,715 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:16,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:16,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:16,716 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:16,716 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:16,716 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:16,716 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:16,716 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:16,716 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:16,716 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:16,716 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:16,716 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:16,717 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. started... 2023-03-31 16:56:16,717 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. as already flushing 2023-03-31 16:56:16,717 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:16,717 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:16,717 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:16,717 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,717 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:16,717 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:16,718 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:16,718 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:16,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:16,719 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:16,719 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:16,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:16,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:16,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:16,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:16,720 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:16,720 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b7e2545[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:16,720 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:16,720 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:16,721 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:16,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:16,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:16,721 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:16,721 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:16,721 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:16,722 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,722 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:16,722 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:16,723 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:16,723 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,723 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:16,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:16,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:16,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:16,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:16,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:16,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:16,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:16,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:16,725 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:16,725 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:16,725 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:16,725 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:16,725 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:16,725 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:16,728 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:16,728 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:16,728 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:16,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:16,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:16,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:16,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:16,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:16,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:16,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:16,730 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:16,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:16,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:16,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:16,731 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:16,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:16,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:16,733 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:16,733 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:16,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:16,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:16,733 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:16,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:16,733 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:16,733 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:16,733 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:16,733 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:16,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:16,733 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:56:16,733 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:16,733 DEBUG [Thread-4140] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:16,734 DEBUG [Thread-4140] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:16,734 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:16,734 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:16,834 DEBUG [Thread-4140] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:16,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:16,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:16,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:16,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:16,837 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:16,837 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:16,837 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:16,837 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:16,838 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:16,838 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:16,841 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:16,842 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:16,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:16,842 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,842 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:16,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:16,842 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:16,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:16,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:16,843 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:16,843 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:16,843 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:16,843 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:16,843 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:16,843 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:16,843 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:16,843 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:16,844 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:16,844 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. started... 2023-03-31 16:56:16,844 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. as already flushing 2023-03-31 16:56:16,844 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:16,844 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:16,844 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:16,844 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,844 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:16,844 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:16,846 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,846 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:16,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:16,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:16,847 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:16,847 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:16,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:16,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:16,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:16,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:16,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:16,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@480877b0[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:16,849 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:16,849 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:16,850 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:16,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:16,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:16,850 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:16,850 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:16,850 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:16,850 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,850 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:16,852 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,852 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:16,852 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:16,852 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:16,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:16,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:16,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:16,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:16,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:16,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:16,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:16,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:16,854 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:16,854 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:16,855 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:16,855 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:16,855 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:16,855 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:16,856 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:16,856 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:16,856 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:16,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:16,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:16,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:16,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:16,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:16,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:16,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:16,857 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:16,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:16,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:16,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:16,858 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:16,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:16,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:16,861 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:16,861 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,861 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:16,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:16,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:16,862 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:16,862 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:16,861 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:16,861 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:16,862 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:16,862 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:16,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:16,862 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:56:16,862 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:16,862 DEBUG [Thread-4140] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:16,862 DEBUG [Thread-4140] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:16,862 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:16,862 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:16,892 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=187 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/B/1228fe929d5143a48e5835667413cce2 2023-03-31 16:56:16,962 DEBUG [Thread-4140] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:16,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:16,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:16,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:16,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:16,966 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:16,966 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:16,966 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:16,966 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:16,966 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:16,966 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:16,968 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,968 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:16,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:16,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:16,968 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,968 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:16,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:16,969 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:16,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:16,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:16,969 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:16,969 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:16,969 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:16,969 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:16,969 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:16,969 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:16,970 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:16,970 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:16,970 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:16,970 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. started... 2023-03-31 16:56:16,970 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. as already flushing 2023-03-31 16:56:16,970 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:16,970 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:16,970 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:16,970 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,970 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:16,970 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:16,971 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:16,971 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:16,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:16,972 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:16,972 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:16,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:16,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:16,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:16,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:16,974 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:16,974 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7221e784[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:16,974 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:16,974 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:16,975 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:16,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:16,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:16,975 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:16,975 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:16,975 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,975 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:16,975 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:16,978 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,978 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:16,978 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:16,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:16,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:16,978 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:16,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:16,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:16,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:16,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:16,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:16,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:16,980 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:16,980 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:16,980 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:16,980 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:16,980 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:16,980 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:16,982 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:16,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:16,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:16,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:16,982 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:16,982 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:16,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:16,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:16,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:16,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:16,982 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:16,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:16,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:16,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:16,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:16,984 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:16,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:16,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,986 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:16,986 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:16,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:16,986 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:16,986 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:16,986 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:16,986 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:56:16,986 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:16,986 DEBUG [Thread-4140] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:16,986 DEBUG [Thread-4140] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:16,986 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:16,986 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:16,986 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:16,987 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:16,987 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:16,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:16,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:16,987 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:17,086 DEBUG [Thread-4140] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:17,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:17,088 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:17,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:17,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:17,090 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:17,090 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:17,090 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:17,090 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:17,090 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:17,090 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:17,091 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:17,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:17,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:17,092 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:17,092 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:17,092 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:17,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:17,092 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:17,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:17,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:17,093 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:17,093 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:17,093 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:17,093 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:17,093 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:17,093 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:17,093 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:17,093 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:17,093 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:17,093 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. started... 2023-03-31 16:56:17,094 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. as already flushing 2023-03-31 16:56:17,094 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:17,094 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:17,094 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:17,094 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:17,094 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:17,094 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:17,096 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:17,096 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:17,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:17,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:17,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:17,096 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:17,096 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:17,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:17,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:17,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:17,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:17,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:17,097 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:17,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7104185e[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:17,098 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:17,098 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:17,099 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:17,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:17,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:17,099 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:17,099 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:17,099 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:17,099 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:17,099 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:17,100 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:17,100 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:17,100 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:17,100 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:17,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:17,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:17,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:17,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:17,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:17,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:17,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:17,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:17,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:17,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:17,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:17,102 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:17,102 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:17,103 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:17,103 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:17,103 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:17,103 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:17,104 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:17,104 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:17,104 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:17,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:17,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:17,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:17,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:17,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:17,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:17,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:17,105 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:17,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:17,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:17,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:17,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:17,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:17,106 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:17,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:17,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:17,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:17,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:17,109 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:17,109 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:17,109 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:17,109 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:17,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:17,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:17,109 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:17,109 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:17,109 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:17,109 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:17,109 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:17,109 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:17,109 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:17,109 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:56:17,109 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:17,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:17,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:17,109 DEBUG [Thread-4140] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:17,109 DEBUG [Thread-4140] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:17,210 DEBUG [Thread-4140] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:17,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:17,212 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:17,212 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:17,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49618 deadline: 1680281837210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:17,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:17,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:17,213 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:17,213 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:17,213 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:17,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49608 deadline: 1680281837212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:17,213 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:17,214 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:17,214 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:17,214 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:17,215 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:17,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49602 deadline: 1680281837212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:17,215 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:17,215 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:17,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:17,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:17,216 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:17,216 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:17,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:17,216 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:17,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:17,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:17,217 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:17,217 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:17,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49568 deadline: 1680281837215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:17,217 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:17,217 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:17,217 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:17,217 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:17,217 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:17,217 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:17,217 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:17,217 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:17,217 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. started... 2023-03-31 16:56:17,217 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. as already flushing 2023-03-31 16:56:17,218 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:17,218 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:17,218 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:17,218 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:17,218 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:17,218 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:17,219 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:17,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49566 deadline: 1680281837216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:17,219 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:17,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:17,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:17,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:17,219 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:17,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:17,220 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:17,220 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:17,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:17,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:17,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:17,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:17,221 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:17,221 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6bca4f36[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:17,221 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:17,221 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:17,226 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:17,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:17,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:17,226 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:17,226 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:17,226 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:17,226 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:17,226 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:17,228 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:17,228 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:17,228 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:17,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:17,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:17,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:17,228 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:17,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:17,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:17,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:17,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:17,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:17,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:17,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:17,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:17,230 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:17,230 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:17,231 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:17,231 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:17,231 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:17,231 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:17,232 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:17,232 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:17,233 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:17,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:17,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:17,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:17,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:17,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:17,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:17,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:17,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:17,233 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:17,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:17,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:17,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:17,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:17,234 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:17,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:17,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:17,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:17,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:17,237 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:17,237 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:17,237 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:17,237 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:17,237 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:17,237 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:17,237 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:17,237 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:17,237 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:17,237 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:17,237 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:56:17,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:17,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:17,238 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:17,238 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:17,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:17,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:17,238 DEBUG [Thread-4140] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:17,238 DEBUG [Thread-4140] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:17,302 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=187 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/C/d4ff7a2d08564fd09d62080417fee920 2023-03-31 16:56:17,308 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/A/af62b9f582dd4018897c68167000682c as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/af62b9f582dd4018897c68167000682c 2023-03-31 16:56:17,311 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/af62b9f582dd4018897c68167000682c, entries=200, sequenceid=187, filesize=32.1 K 2023-03-31 16:56:17,312 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/B/1228fe929d5143a48e5835667413cce2 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/1228fe929d5143a48e5835667413cce2 2023-03-31 16:56:17,315 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/1228fe929d5143a48e5835667413cce2, entries=150, sequenceid=187, filesize=11.8 K 2023-03-31 16:56:17,315 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/C/d4ff7a2d08564fd09d62080417fee920 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/d4ff7a2d08564fd09d62080417fee920 2023-03-31 16:56:17,318 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/d4ff7a2d08564fd09d62080417fee920, entries=150, sequenceid=187, filesize=11.8 K 2023-03-31 16:56:17,319 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=138.65 KB/141980 for 313b3df11d64a9396698e7102a97394f in 1275ms, sequenceid=187, compaction requested=true 2023-03-31 16:56:17,319 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 313b3df11d64a9396698e7102a97394f: 2023-03-31 16:56:17,319 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:56:17,319 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 16:56:17,319 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 16:56:17,320 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:56:17,320 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:56:17,320 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85020 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 16:56:17,320 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1912): 313b3df11d64a9396698e7102a97394f/A is initiating minor compaction (all files) 2023-03-31 16:56:17,320 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 313b3df11d64a9396698e7102a97394f/A in TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:17,321 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/ee7a65ff8aac4e2baee210c3798c621c, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/2132eedb9a33436ca187dcc7974e49b3, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/af62b9f582dd4018897c68167000682c] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp, totalSize=83.0 K 2023-03-31 16:56:17,321 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48352 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 16:56:17,321 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): 313b3df11d64a9396698e7102a97394f/B is initiating minor compaction (all files) 2023-03-31 16:56:17,321 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 313b3df11d64a9396698e7102a97394f/B in TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:17,321 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/77318e9a09de4e2d8c7cbc694418ceb4, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/b253ed4b4dc84085a4474218345fd11d, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/a433bbefb54840d68735d3333436d8f0, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/1228fe929d5143a48e5835667413cce2] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp, totalSize=47.2 K 2023-03-31 16:56:17,321 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting ee7a65ff8aac4e2baee210c3798c621c, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=163, earliestPutTs=1680281773865 2023-03-31 16:56:17,321 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 77318e9a09de4e2d8c7cbc694418ceb4, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=131, earliestPutTs=1680281773662 2023-03-31 16:56:17,322 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 2132eedb9a33436ca187dcc7974e49b3, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=176, earliestPutTs=1680281773915 2023-03-31 16:56:17,322 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting b253ed4b4dc84085a4474218345fd11d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=163, earliestPutTs=1680281773865 2023-03-31 16:56:17,322 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting af62b9f582dd4018897c68167000682c, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=187, earliestPutTs=1680281776034 2023-03-31 16:56:17,322 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting a433bbefb54840d68735d3333436d8f0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=176, earliestPutTs=1680281773915 2023-03-31 16:56:17,323 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 1228fe929d5143a48e5835667413cce2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=187, earliestPutTs=1680281776034 2023-03-31 16:56:17,332 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] throttle.PressureAwareThroughputController(145): 313b3df11d64a9396698e7102a97394f#A#compaction#599 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:56:17,334 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): 313b3df11d64a9396698e7102a97394f#B#compaction#600 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:56:17,338 DEBUG [Thread-4140] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:17,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:17,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:17,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:17,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:17,352 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:17,352 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:17,352 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:17,352 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:17,353 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:17,353 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:17,355 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:17,355 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:17,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:17,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:17,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:17,356 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:17,356 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:17,356 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:17,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:17,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:17,356 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:17,357 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:17,357 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:17,357 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:17,357 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:17,357 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:17,371 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:17,371 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:17,372 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:17,372 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. started... 2023-03-31 16:56:17,372 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 313b3df11d64a9396698e7102a97394f 3/3 column families, dataSize=138.65 KB heapSize=395.14 KB 2023-03-31 16:56:17,372 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=A 2023-03-31 16:56:17,372 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:56:17,372 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=B 2023-03-31 16:56:17,372 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:56:17,373 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=C 2023-03-31 16:56:17,373 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:56:17,373 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/B/37f223b1c19f4a2b8728c033ec88484a as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/37f223b1c19f4a2b8728c033ec88484a 2023-03-31 16:56:17,377 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 313b3df11d64a9396698e7102a97394f/B of 313b3df11d64a9396698e7102a97394f into 37f223b1c19f4a2b8728c033ec88484a(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:56:17,377 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 313b3df11d64a9396698e7102a97394f: 2023-03-31 16:56:17,377 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f., storeName=313b3df11d64a9396698e7102a97394f/B, priority=12, startTime=1680281777319; duration=0sec 2023-03-31 16:56:17,377 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:56:17,377 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 16:56:17,378 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48352 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 16:56:17,379 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): 313b3df11d64a9396698e7102a97394f/C is initiating minor compaction (all files) 2023-03-31 16:56:17,379 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 313b3df11d64a9396698e7102a97394f/C in TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:17,379 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/c9844faef41c4de88e6af9777b711d5b, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/02c00cbf15df4f7ab7fc69d129eef442, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/435cc93dd6194ef2bc78a80c47d8af90, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/d4ff7a2d08564fd09d62080417fee920] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp, totalSize=47.2 K 2023-03-31 16:56:17,379 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting c9844faef41c4de88e6af9777b711d5b, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=131, earliestPutTs=1680281773662 2023-03-31 16:56:17,380 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 02c00cbf15df4f7ab7fc69d129eef442, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=163, earliestPutTs=1680281773865 2023-03-31 16:56:17,380 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 435cc93dd6194ef2bc78a80c47d8af90, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=176, earliestPutTs=1680281773915 2023-03-31 16:56:17,381 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting d4ff7a2d08564fd09d62080417fee920, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=187, earliestPutTs=1680281776034 2023-03-31 16:56:17,401 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): 313b3df11d64a9396698e7102a97394f#C#compaction#602 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:56:17,416 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303310d1b991189384d0290a4d4a84ad420f4 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303310d1b991189384d0290a4d4a84ad420f4 2023-03-31 16:56:17,426 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=225, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/A/3410bb7b47364241a0b2f85cd2382136 2023-03-31 16:56:17,429 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/C/dac5c4273b9e4ef8a74f0382cda66cff as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/dac5c4273b9e4ef8a74f0382cda66cff 2023-03-31 16:56:17,434 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 313b3df11d64a9396698e7102a97394f/C of 313b3df11d64a9396698e7102a97394f into dac5c4273b9e4ef8a74f0382cda66cff(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:56:17,434 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 313b3df11d64a9396698e7102a97394f: 2023-03-31 16:56:17,434 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f., storeName=313b3df11d64a9396698e7102a97394f/C, priority=12, startTime=1680281777320; duration=0sec 2023-03-31 16:56:17,434 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:56:17,801 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/A/18e62844b9d6410387523e7afef6d554 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/18e62844b9d6410387523e7afef6d554 2023-03-31 16:56:17,805 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 313b3df11d64a9396698e7102a97394f/A of 313b3df11d64a9396698e7102a97394f into 18e62844b9d6410387523e7afef6d554(size=25.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:56:17,805 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 313b3df11d64a9396698e7102a97394f: 2023-03-31 16:56:17,805 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f., storeName=313b3df11d64a9396698e7102a97394f/A, priority=13, startTime=1680281777319; duration=0sec 2023-03-31 16:56:17,805 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:56:17,842 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=225 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/B/38084459697d4468be82f5c75bfd1779 2023-03-31 16:56:17,852 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=225 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/C/d36534d94d3247f7a39e082d59684576 2023-03-31 16:56:17,856 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/A/3410bb7b47364241a0b2f85cd2382136 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/3410bb7b47364241a0b2f85cd2382136 2023-03-31 16:56:17,859 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/3410bb7b47364241a0b2f85cd2382136, entries=150, sequenceid=225, filesize=25.3 K 2023-03-31 16:56:17,860 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/B/38084459697d4468be82f5c75bfd1779 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/38084459697d4468be82f5c75bfd1779 2023-03-31 16:56:17,863 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/38084459697d4468be82f5c75bfd1779, entries=150, sequenceid=225, filesize=11.8 K 2023-03-31 16:56:17,864 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/C/d36534d94d3247f7a39e082d59684576 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/d36534d94d3247f7a39e082d59684576 2023-03-31 16:56:17,867 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/d36534d94d3247f7a39e082d59684576, entries=150, sequenceid=225, filesize=11.8 K 2023-03-31 16:56:17,868 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~138.65 KB/141980, heapSize ~395.09 KB/404576, currentSize=0 B/0 for 313b3df11d64a9396698e7102a97394f in 496ms, sequenceid=225, compaction requested=false 2023-03-31 16:56:17,868 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 313b3df11d64a9396698e7102a97394f: 2023-03-31 16:56:17,868 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:17,868 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:17,868 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:17,868 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:17,868 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:17,868 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:17,870 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:17,870 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:17,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:17,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:17,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:17,870 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:17,871 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:17,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:17,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:17,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:17,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:17,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:17,872 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:17,872 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60741008[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:17,872 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:17,872 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:17,874 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:17,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:17,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:17,874 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:17,874 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:17,874 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:17,874 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:17,874 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:17,875 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:17,875 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:17,875 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:17,875 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:17,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:17,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:17,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:17,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:17,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:17,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:17,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:17,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:17,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:17,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:17,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:17,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:17,878 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:17,878 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:17,878 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:17,878 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:17,878 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:17,881 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:17,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:17,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:17,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:17,881 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:17,881 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:17,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:17,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:17,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:17,881 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:17,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:17,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:17,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:17,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:17,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:17,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:17,882 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:17,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:17,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:17,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:17,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:17,885 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:17,885 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:17,885 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:17,885 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:17,885 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:17,885 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:17,885 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:17,885 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:17,885 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:56:17,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:17,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:17,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:17,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:17,885 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:17,885 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:17,885 DEBUG [Thread-4140] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:17,886 DEBUG [Thread-4140] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:17,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:17,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:17,986 DEBUG [Thread-4140] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:17,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:17,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:17,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:17,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:17,989 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:17,989 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:17,989 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:17,989 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:17,990 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:17,990 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:17,991 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:17,991 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:17,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:17,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:17,991 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:17,991 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:17,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:17,992 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:17,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:17,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:17,992 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:17,992 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:17,992 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:17,993 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:17,993 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:17,993 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:17,993 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:17,993 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:17,993 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:17,993 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. started... 2023-03-31 16:56:17,993 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 313b3df11d64a9396698e7102a97394f: 2023-03-31 16:56:17,993 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:17,993 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:17,993 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:17,993 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:17,993 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:17,993 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:17,995 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:17,995 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:17,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:17,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:17,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:17,995 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:17,995 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:17,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:17,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:17,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:17,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:17,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:17,997 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:17,997 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d14d645[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:17,997 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:17,997 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:18,001 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:18,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:18,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:18,002 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:18,002 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:18,002 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:18,002 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,002 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:18,003 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,003 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:18,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:18,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:18,003 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:18,003 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:18,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:18,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:18,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:18,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:18,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:18,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:18,006 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:18,006 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:18,006 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:18,006 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:18,006 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:18,006 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:18,008 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:18,008 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:18,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:18,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:18,008 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:18,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:18,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:18,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:18,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:18,008 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:18,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:18,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:18,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:18,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:18,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:18,009 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:18,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:18,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,012 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,012 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:18,012 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:18,012 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,012 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:18,012 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:18,012 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:18,012 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:18,012 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:18,012 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:18,012 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:56:18,012 DEBUG [Thread-4140] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:18,012 DEBUG [Thread-4140] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:18,012 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:18,013 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:18,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:18,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:18,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:18,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:18,113 DEBUG [Thread-4140] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:18,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:18,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:18,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:18,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:18,116 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:18,116 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:18,116 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:18,116 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:18,116 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:18,116 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:18,119 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:18,119 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:18,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:18,119 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,119 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:18,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:18,120 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:18,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:18,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:18,120 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:18,121 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:18,121 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:18,121 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:18,121 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:18,121 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:18,121 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:18,121 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:18,121 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:18,121 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. started... 2023-03-31 16:56:18,121 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 313b3df11d64a9396698e7102a97394f: 2023-03-31 16:56:18,121 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:18,121 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:18,121 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:18,121 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,121 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:18,122 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:18,123 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:18,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:18,123 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:18,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:18,124 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:18,124 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:18,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:18,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:18,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:18,125 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:18,125 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c83eb51[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:18,125 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:18,125 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:18,126 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:18,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:18,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:18,127 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:18,127 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:18,127 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:18,127 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,127 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:18,131 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:18,131 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:18,131 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,131 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:18,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:18,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:18,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:18,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:18,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:18,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:18,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:18,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:18,133 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:18,133 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:18,133 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:18,133 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:18,133 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:18,134 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:18,135 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:18,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:18,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:18,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:18,135 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:18,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:18,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:18,135 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:18,136 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:18,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:18,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:18,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:18,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:18,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:18,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:18,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,137 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:18,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:18,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,140 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:18,140 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:18,140 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,140 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:18,140 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:18,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:18,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:18,140 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:56:18,140 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,140 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:18,140 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:18,140 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:18,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:18,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:18,140 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:18,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:18,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:18,141 DEBUG [Thread-4140] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:18,141 DEBUG [Thread-4140] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:18,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on 313b3df11d64a9396698e7102a97394f 2023-03-31 16:56:18,240 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 313b3df11d64a9396698e7102a97394f 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-31 16:56:18,241 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=A 2023-03-31 16:56:18,241 DEBUG [Thread-4140] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:18,241 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:56:18,241 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=B 2023-03-31 16:56:18,241 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:56:18,241 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=C 2023-03-31 16:56:18,241 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:56:18,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:18,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:18,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:18,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:18,244 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:18,244 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:18,244 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:18,244 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:18,246 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:18,246 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:18,250 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:18,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:18,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:18,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:18,250 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:18,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:18,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:18,251 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:18,252 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,252 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,252 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:18,259 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:18,259 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:18,259 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:18,260 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:18,260 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:18,260 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:18,260 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:18,260 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:18,260 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. started... 2023-03-31 16:56:18,260 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. as already flushing 2023-03-31 16:56:18,260 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:18,260 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:18,260 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:18,260 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,260 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:18,260 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:18,261 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680281778251, max timestamp=1680281778259Num uniques 859; 2023-03-31 16:56:18,262 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 16:56:18,262 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680281778251, max timestamp=1680281778259Num uniques 859; 2023-03-31 16:56:18,262 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 16:56:18,262 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680281778251, max timestamp=1680281778259Num uniques 859; 2023-03-31 16:56:18,262 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 16:56:18,264 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:18,264 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:18,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:18,264 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:18,264 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:18,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:18,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:18,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:18,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:18,266 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:18,266 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7589be08[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:18,266 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:18,266 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:18,267 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:18,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:18,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:18,267 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:18,267 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:18,268 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,268 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:18,268 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:18,273 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,273 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:18,273 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:18,273 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:18,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:18,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:18,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:18,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:18,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:18,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:18,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:18,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:18,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:18,277 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:18,277 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:18,277 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:18,277 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:18,277 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:18,280 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:18,280 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:18,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:18,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:18,280 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:18,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:18,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:18,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:18,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:18,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:18,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:18,282 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,283 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,284 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:18,285 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:18,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:18,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:18,285 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:18,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49608 deadline: 1680281838279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:18,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:18,286 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,286 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:18,286 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:18,286 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:18,286 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,286 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:18,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:18,287 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:18,286 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:18,287 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:18,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49618 deadline: 1680281838282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,287 DEBUG [Thread-4140] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:18,287 DEBUG [Thread-4140] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:18,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49568 deadline: 1680281838282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,286 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:18,288 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:18,288 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:18,287 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:18,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49566 deadline: 1680281838282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:18,289 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:18,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49602 deadline: 1680281838286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:18,290 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:56:18,387 DEBUG [Thread-4140] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:18,388 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:18,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:18,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49608 deadline: 1680281838386, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,390 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:18,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:18,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:18,391 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:18,391 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:18,391 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:18,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49618 deadline: 1680281838389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,391 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:18,391 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:18,391 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:18,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49568 deadline: 1680281838389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,392 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:18,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49566 deadline: 1680281838389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,392 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:18,392 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:18,392 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:18,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49602 deadline: 1680281838390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,393 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:18,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:18,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:18,393 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:18,393 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,394 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:18,394 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:18,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:18,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:18,394 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:18,394 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:18,394 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:18,394 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:18,394 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:18,394 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:18,395 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:18,395 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:18,395 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:18,395 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. started... 2023-03-31 16:56:18,395 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. as already flushing 2023-03-31 16:56:18,395 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:18,395 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:18,395 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:18,395 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,395 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:18,395 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:18,397 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,397 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:18,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:18,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:18,397 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:18,397 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:18,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:18,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:18,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:18,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:18,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:18,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@353d14e1[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:18,398 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:18,399 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:18,401 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:18,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:18,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:18,401 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:18,401 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:18,401 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:18,401 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,401 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:18,402 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:18,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:18,403 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:18,403 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:18,403 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:18,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:18,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:18,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:18,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:18,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:18,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:18,409 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:18,409 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:18,409 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:18,409 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:18,409 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:18,409 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:18,411 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:18,411 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:18,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:18,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:18,412 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,412 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:18,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:18,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:18,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:18,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:18,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:18,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:18,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:18,414 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:18,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:18,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:18,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:18,418 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:18,418 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:18,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:18,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:18,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:18,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:18,418 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:18,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:18,419 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:18,418 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,419 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:18,419 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:18,419 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,419 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:18,419 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:18,419 DEBUG [Thread-4140] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:18,419 DEBUG [Thread-4140] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:18,419 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:18,419 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:56:18,519 DEBUG [Thread-4140] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:18,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:18,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:18,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:18,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:18,522 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:18,522 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:18,522 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:18,522 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:18,523 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:18,523 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:18,524 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,524 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:18,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:18,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:18,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:18,524 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,524 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:18,525 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:18,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:18,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:18,525 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:18,525 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:18,525 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:18,525 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:18,525 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:18,525 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:18,526 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:18,526 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:18,526 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:18,526 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. started... 2023-03-31 16:56:18,526 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. as already flushing 2023-03-31 16:56:18,526 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:18,526 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:18,526 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:18,526 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,526 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:18,526 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:18,527 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:18,527 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:18,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:18,528 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:18,528 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:18,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:18,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:18,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:18,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:18,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:18,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ea04afd[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:18,529 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:18,529 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:18,530 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:18,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:18,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:18,530 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:18,530 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:18,531 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:18,531 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,531 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:18,533 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,533 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:18,533 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:18,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:18,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:18,533 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:18,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:18,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:18,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:18,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:18,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:18,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:18,535 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:18,535 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:18,535 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:18,535 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:18,535 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:18,535 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:18,537 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:18,537 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:18,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:18,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:18,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:18,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:18,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:18,537 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:18,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:18,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:18,537 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:18,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:18,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:18,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:18,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:18,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:18,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:18,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,540 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,545 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,545 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:18,545 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:18,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:18,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:18,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:18,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:18,545 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:18,545 DEBUG [Thread-4140] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:18,545 DEBUG [Thread-4140] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:18,545 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:18,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:18,545 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,545 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:18,545 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:18,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:18,545 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:18,545 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:18,591 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:18,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49608 deadline: 1680281838589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,595 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:18,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49568 deadline: 1680281838592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,595 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:18,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49618 deadline: 1680281838593, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,596 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:18,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49566 deadline: 1680281838593, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,596 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:18,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49602 deadline: 1680281838594, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,645 DEBUG [Thread-4140] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:18,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:18,647 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:18,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:18,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:18,648 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:18,648 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:18,648 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:18,648 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:18,649 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:18,649 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:18,650 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:18,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:18,650 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:18,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:18,650 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,650 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:18,651 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:18,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:18,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:18,651 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:18,651 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:18,651 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:18,651 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:18,651 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:18,652 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:18,652 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:18,652 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:18,652 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:18,652 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. started... 2023-03-31 16:56:18,652 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. as already flushing 2023-03-31 16:56:18,652 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:18,652 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:18,652 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:18,652 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,652 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:18,652 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:18,654 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,654 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:18,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:18,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:18,654 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:18,654 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:18,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:18,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:18,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:18,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:18,655 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:18,655 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5daaf11b[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:18,655 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:18,655 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:18,657 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:18,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:18,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:18,657 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,658 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:18,658 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:18,658 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:18,658 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:18,659 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,659 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:18,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:18,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:18,659 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:18,660 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:18,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:18,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:18,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:18,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:18,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:18,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:18,662 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:18,662 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:18,662 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:18,662 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:18,662 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:18,662 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:18,664 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:18,664 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:18,664 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:18,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:18,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:18,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:18,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:18,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:18,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:18,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:18,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:18,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:18,665 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:18,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:18,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:18,666 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,667 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230331c95713d984a449f0b7231e0297e465ec to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331c95713d984a449f0b7231e0297e465ec 2023-03-31 16:56:18,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:18,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:18,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,670 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:18,670 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:18,670 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:18,670 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:18,670 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:18,671 DEBUG [Thread-4140] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:18,671 DEBUG [Thread-4140] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:18,671 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,671 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:18,671 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:18,671 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,671 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:18,671 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:18,671 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:18,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:18,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:18,671 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:56:18,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:18,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:18,679 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=241, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/A/a42fb449377441b0a1c54e26cc0af92b 2023-03-31 16:56:18,687 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=241 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/B/f4cc6521410c4a4ab11f6107ef25dd0c 2023-03-31 16:56:18,696 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=241 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/C/78344cc17bda41639d06cbd5c7f81eec 2023-03-31 16:56:18,699 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/A/a42fb449377441b0a1c54e26cc0af92b as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/a42fb449377441b0a1c54e26cc0af92b 2023-03-31 16:56:18,702 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/a42fb449377441b0a1c54e26cc0af92b, entries=400, sequenceid=241, filesize=59.4 K 2023-03-31 16:56:18,703 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/B/f4cc6521410c4a4ab11f6107ef25dd0c as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/f4cc6521410c4a4ab11f6107ef25dd0c 2023-03-31 16:56:18,705 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/f4cc6521410c4a4ab11f6107ef25dd0c, entries=150, sequenceid=241, filesize=11.8 K 2023-03-31 16:56:18,706 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/C/78344cc17bda41639d06cbd5c7f81eec as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/78344cc17bda41639d06cbd5c7f81eec 2023-03-31 16:56:18,710 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/78344cc17bda41639d06cbd5c7f81eec, entries=150, sequenceid=241, filesize=11.8 K 2023-03-31 16:56:18,710 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 313b3df11d64a9396698e7102a97394f in 470ms, sequenceid=241, compaction requested=true 2023-03-31 16:56:18,710 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 313b3df11d64a9396698e7102a97394f: 2023-03-31 16:56:18,711 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:56:18,711 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 16:56:18,711 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 16:56:18,711 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:56:18,711 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:56:18,712 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 113090 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 16:56:18,712 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1912): 313b3df11d64a9396698e7102a97394f/A is initiating minor compaction (all files) 2023-03-31 16:56:18,712 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 313b3df11d64a9396698e7102a97394f/A in TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:18,712 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/18e62844b9d6410387523e7afef6d554, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/3410bb7b47364241a0b2f85cd2382136, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/a42fb449377441b0a1c54e26cc0af92b] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp, totalSize=110.4 K 2023-03-31 16:56:18,712 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36552 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 16:56:18,712 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): 313b3df11d64a9396698e7102a97394f/B is initiating minor compaction (all files) 2023-03-31 16:56:18,712 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 313b3df11d64a9396698e7102a97394f/B in TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:18,712 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/37f223b1c19f4a2b8728c033ec88484a, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/38084459697d4468be82f5c75bfd1779, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/f4cc6521410c4a4ab11f6107ef25dd0c] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp, totalSize=35.7 K 2023-03-31 16:56:18,713 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 37f223b1c19f4a2b8728c033ec88484a, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=187, earliestPutTs=1680281776034 2023-03-31 16:56:18,713 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 18e62844b9d6410387523e7afef6d554, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=187, earliestPutTs=1680281776034 2023-03-31 16:56:18,713 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 38084459697d4468be82f5c75bfd1779, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=225, earliestPutTs=1680281776090 2023-03-31 16:56:18,713 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 3410bb7b47364241a0b2f85cd2382136, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=225, earliestPutTs=1680281776090 2023-03-31 16:56:18,714 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting f4cc6521410c4a4ab11f6107ef25dd0c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=241, earliestPutTs=1680281778232 2023-03-31 16:56:18,714 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting a42fb449377441b0a1c54e26cc0af92b, keycount=400, bloomtype=ROW, size=59.4 K, encoding=NONE, compression=NONE, seqNum=241, earliestPutTs=1680281778226 2023-03-31 16:56:18,721 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): 313b3df11d64a9396698e7102a97394f#B#compaction#608 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:56:18,733 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] throttle.PressureAwareThroughputController(145): 313b3df11d64a9396698e7102a97394f#A#compaction#609 average throughput is 1.79 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:56:18,737 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/B/5f1ff6b263554c819345d606c6ed8122 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/5f1ff6b263554c819345d606c6ed8122 2023-03-31 16:56:18,742 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 313b3df11d64a9396698e7102a97394f/B of 313b3df11d64a9396698e7102a97394f into 5f1ff6b263554c819345d606c6ed8122(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:56:18,742 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 313b3df11d64a9396698e7102a97394f: 2023-03-31 16:56:18,742 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f., storeName=313b3df11d64a9396698e7102a97394f/B, priority=13, startTime=1680281778711; duration=0sec 2023-03-31 16:56:18,742 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:56:18,742 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 16:56:18,743 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36552 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 16:56:18,743 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): 313b3df11d64a9396698e7102a97394f/C is initiating minor compaction (all files) 2023-03-31 16:56:18,743 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 313b3df11d64a9396698e7102a97394f/C in TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:18,743 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/dac5c4273b9e4ef8a74f0382cda66cff, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/d36534d94d3247f7a39e082d59684576, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/78344cc17bda41639d06cbd5c7f81eec] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp, totalSize=35.7 K 2023-03-31 16:56:18,744 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting dac5c4273b9e4ef8a74f0382cda66cff, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=187, earliestPutTs=1680281776034 2023-03-31 16:56:18,744 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting d36534d94d3247f7a39e082d59684576, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=225, earliestPutTs=1680281776090 2023-03-31 16:56:18,744 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 78344cc17bda41639d06cbd5c7f81eec, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=241, earliestPutTs=1680281778232 2023-03-31 16:56:18,749 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): 313b3df11d64a9396698e7102a97394f#C#compaction#610 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:56:18,757 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/C/fcf6fe37d91e4ac49edfeb7d1761ee77 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/fcf6fe37d91e4ac49edfeb7d1761ee77 2023-03-31 16:56:18,761 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 313b3df11d64a9396698e7102a97394f/C of 313b3df11d64a9396698e7102a97394f into fcf6fe37d91e4ac49edfeb7d1761ee77(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:56:18,761 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 313b3df11d64a9396698e7102a97394f: 2023-03-31 16:56:18,761 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f., storeName=313b3df11d64a9396698e7102a97394f/C, priority=13, startTime=1680281778711; duration=0sec 2023-03-31 16:56:18,761 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:56:18,771 DEBUG [Thread-4140] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:18,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:18,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:18,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:18,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:18,774 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:18,774 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:18,774 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:18,774 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:18,774 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:18,774 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:18,777 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:18,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:18,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:18,777 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:18,777 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,777 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:18,777 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:18,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:18,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:18,778 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:18,778 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:18,778 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:18,778 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:18,778 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:18,778 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:18,778 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:18,779 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:18,779 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:18,779 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. started... 2023-03-31 16:56:18,779 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 313b3df11d64a9396698e7102a97394f 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-03-31 16:56:18,779 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=A 2023-03-31 16:56:18,779 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:56:18,779 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=B 2023-03-31 16:56:18,779 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:56:18,779 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=C 2023-03-31 16:56:18,779 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:56:18,801 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023033171ba1280c2864d7fbdd1ab775c218805 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033171ba1280c2864d7fbdd1ab775c218805 2023-03-31 16:56:18,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on 313b3df11d64a9396698e7102a97394f 2023-03-31 16:56:18,894 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. as already flushing 2023-03-31 16:56:18,906 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:18,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49608 deadline: 1680281838902, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,906 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:18,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49602 deadline: 1680281838902, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,906 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:18,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49618 deadline: 1680281838903, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,908 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:18,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49568 deadline: 1680281838905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:18,910 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:18,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49566 deadline: 1680281838906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:19,008 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:19,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49608 deadline: 1680281839007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:19,008 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:19,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49618 deadline: 1680281839007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:19,008 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:19,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49602 deadline: 1680281839007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:19,012 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:19,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49568 deadline: 1680281839009, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:19,012 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:19,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49566 deadline: 1680281839011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:19,150 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/A/04ce8f4061424fa487158be331c9e02d as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/04ce8f4061424fa487158be331c9e02d 2023-03-31 16:56:19,154 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 313b3df11d64a9396698e7102a97394f/A of 313b3df11d64a9396698e7102a97394f into 04ce8f4061424fa487158be331c9e02d(size=25.8 K), total size for store is 25.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:56:19,154 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 313b3df11d64a9396698e7102a97394f: 2023-03-31 16:56:19,154 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f., storeName=313b3df11d64a9396698e7102a97394f/A, priority=13, startTime=1680281778710; duration=0sec 2023-03-31 16:56:19,154 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:56:19,211 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:19,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49618 deadline: 1680281839209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:19,213 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:19,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49608 deadline: 1680281839210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:19,213 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:19,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49602 deadline: 1680281839210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:19,214 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:19,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49566 deadline: 1680281839213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:19,216 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:19,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49568 deadline: 1680281839214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:19,217 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=278, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/A/dd1d4fe0b0b64966be099cb76c7aded0 2023-03-31 16:56:19,227 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=278 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/B/afe5223c9fc442118374543638e2af09 2023-03-31 16:56:19,236 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=278 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/C/9ca45eb906334f76b38464b9659bc024 2023-03-31 16:56:19,239 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/A/dd1d4fe0b0b64966be099cb76c7aded0 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/dd1d4fe0b0b64966be099cb76c7aded0 2023-03-31 16:56:19,243 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/dd1d4fe0b0b64966be099cb76c7aded0, entries=150, sequenceid=278, filesize=25.4 K 2023-03-31 16:56:19,243 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/B/afe5223c9fc442118374543638e2af09 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/afe5223c9fc442118374543638e2af09 2023-03-31 16:56:19,247 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/afe5223c9fc442118374543638e2af09, entries=150, sequenceid=278, filesize=11.9 K 2023-03-31 16:56:19,248 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/C/9ca45eb906334f76b38464b9659bc024 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/9ca45eb906334f76b38464b9659bc024 2023-03-31 16:56:19,251 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/9ca45eb906334f76b38464b9659bc024, entries=150, sequenceid=278, filesize=11.9 K 2023-03-31 16:56:19,252 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for 313b3df11d64a9396698e7102a97394f in 473ms, sequenceid=278, compaction requested=false 2023-03-31 16:56:19,252 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 313b3df11d64a9396698e7102a97394f: 2023-03-31 16:56:19,252 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:19,252 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:19,252 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:19,252 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:19,252 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:19,252 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:19,255 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:19,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:19,255 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:19,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:19,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:19,255 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:19,255 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:19,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:19,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:19,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:19,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:19,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:19,257 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:19,257 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65eb6d58[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:19,257 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:19,257 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:19,259 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:19,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:19,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:19,259 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:19,259 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:19,259 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:19,259 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:19,259 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:19,260 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:19,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:19,260 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:19,261 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:19,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:19,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:19,261 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:19,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:19,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:19,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:19,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:19,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:19,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:19,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:19,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:19,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:19,263 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:19,263 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:19,263 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:19,263 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:19,263 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:19,265 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:19,265 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:19,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:19,265 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:19,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:19,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:19,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:19,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:19,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:19,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:19,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:19,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:19,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:19,266 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:19,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:19,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:19,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:19,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:19,267 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:19,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:19,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:19,269 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:19,269 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:19,269 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:19,269 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:19,269 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:19,269 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:19,269 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:19,269 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:19,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:19,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:19,270 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:19,269 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:19,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:19,270 DEBUG [Thread-4140] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:19,270 DEBUG [Thread-4140] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:19,269 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:19,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:19,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:19,370 DEBUG [Thread-4140] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:19,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:19,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:19,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:19,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:19,373 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:19,373 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:19,373 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:19,373 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:19,373 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:19,373 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:19,375 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:19,375 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:19,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:19,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:19,375 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:19,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:19,375 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:19,375 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:19,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:19,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:19,376 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:19,376 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:19,376 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:19,376 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:19,376 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:19,376 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:19,377 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:19,377 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:19,377 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:19,377 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. started... 2023-03-31 16:56:19,377 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 313b3df11d64a9396698e7102a97394f 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-31 16:56:19,377 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=A 2023-03-31 16:56:19,377 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:56:19,377 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=B 2023-03-31 16:56:19,377 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:56:19,377 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=C 2023-03-31 16:56:19,377 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:56:19,393 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303310db1873573794e1b9967bba088b428a1 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303310db1873573794e1b9967bba088b428a1 2023-03-31 16:56:19,402 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=290, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/A/5573c6fa554946628ea04eeaa68e9da2 2023-03-31 16:56:19,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on 313b3df11d64a9396698e7102a97394f 2023-03-31 16:56:19,514 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. as already flushing 2023-03-31 16:56:19,535 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680281779520, max timestamp=1680281779533Num uniques 859; 2023-03-31 16:56:19,535 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680281779520, max timestamp=1680281779533Num uniques 859; 2023-03-31 16:56:19,535 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 16:56:19,535 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680281779520, max timestamp=1680281779533Num uniques 859; 2023-03-31 16:56:19,535 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 16:56:19,535 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 16:56:19,551 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:19,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49618 deadline: 1680281839547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:19,551 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:19,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49566 deadline: 1680281839547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:19,552 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:19,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49608 deadline: 1680281839548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:19,553 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:19,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49568 deadline: 1680281839549, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:19,553 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:19,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49602 deadline: 1680281839549, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:19,654 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:19,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49566 deadline: 1680281839652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:19,655 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:19,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49618 deadline: 1680281839652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:19,655 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:19,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49608 deadline: 1680281839652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:19,655 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:19,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49568 deadline: 1680281839654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:19,656 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:19,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49602 deadline: 1680281839654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:19,812 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=290 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/B/beff2d13f7ad418e8cd8a51bfc0b119b 2023-03-31 16:56:19,858 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:19,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49608 deadline: 1680281839856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:19,858 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:19,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49568 deadline: 1680281839856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:19,859 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:19,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49618 deadline: 1680281839856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:19,859 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:19,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49566 deadline: 1680281839857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:19,859 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:19,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49602 deadline: 1680281839857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,162 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:20,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49568 deadline: 1680281840160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,163 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:20,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49608 deadline: 1680281840161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,163 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:20,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49618 deadline: 1680281840161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,163 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:20,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49566 deadline: 1680281840162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,164 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:20,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49602 deadline: 1680281840163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,222 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=290 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/C/9b743e44bca748e19b9090ccd77cacbb 2023-03-31 16:56:20,226 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/A/5573c6fa554946628ea04eeaa68e9da2 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/5573c6fa554946628ea04eeaa68e9da2 2023-03-31 16:56:20,230 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/5573c6fa554946628ea04eeaa68e9da2, entries=150, sequenceid=290, filesize=25.4 K 2023-03-31 16:56:20,231 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/B/beff2d13f7ad418e8cd8a51bfc0b119b as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/beff2d13f7ad418e8cd8a51bfc0b119b 2023-03-31 16:56:20,234 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/beff2d13f7ad418e8cd8a51bfc0b119b, entries=150, sequenceid=290, filesize=11.9 K 2023-03-31 16:56:20,235 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/C/9b743e44bca748e19b9090ccd77cacbb as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/9b743e44bca748e19b9090ccd77cacbb 2023-03-31 16:56:20,238 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/9b743e44bca748e19b9090ccd77cacbb, entries=150, sequenceid=290, filesize=11.9 K 2023-03-31 16:56:20,238 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 313b3df11d64a9396698e7102a97394f in 861ms, sequenceid=290, compaction requested=true 2023-03-31 16:56:20,239 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 313b3df11d64a9396698e7102a97394f: 2023-03-31 16:56:20,239 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:20,239 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:20,239 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:20,239 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,239 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:20,239 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:20,240 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:20,240 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:20,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:20,241 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:20,241 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:20,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:20,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:20,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:20,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:20,242 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:20,242 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38acf718[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:20,242 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:20,242 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:20,244 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:20,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:20,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:20,245 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:20,245 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:20,245 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:20,245 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,245 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:20,246 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:20,246 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,246 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:20,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:20,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:20,246 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:20,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:20,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:20,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:20,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:20,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:20,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:20,249 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:20,249 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:20,249 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:20,249 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:20,249 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:20,249 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:20,250 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:20,250 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:20,250 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:20,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:20,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:20,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:20,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:20,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:20,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:20,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:20,251 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:20,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:20,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:20,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:20,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:20,252 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:20,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:20,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,257 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,258 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:20,258 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:20,258 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,258 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:20,258 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:20,258 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:20,258 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:56:20,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:20,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:20,258 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:20,258 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:20,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:20,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:20,258 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:20,258 DEBUG [Thread-4140] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:20,258 DEBUG [Thread-4140] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:20,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:20,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:20,358 DEBUG [Thread-4140] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:20,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:20,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:20,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:20,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:20,362 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:20,362 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:20,362 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:20,362 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:20,362 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:20,362 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:20,364 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:20,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:20,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:20,364 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,364 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,364 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:20,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:20,365 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:20,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:20,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:20,365 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:20,365 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:20,365 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:20,365 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:20,365 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:20,365 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:20,366 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:20,366 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:20,366 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:20,366 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. started... 2023-03-31 16:56:20,366 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 313b3df11d64a9396698e7102a97394f 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-03-31 16:56:20,366 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=A 2023-03-31 16:56:20,366 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:56:20,366 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=B 2023-03-31 16:56:20,366 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:56:20,366 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=C 2023-03-31 16:56:20,366 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:56:20,384 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303313f7eb0825229442a8e787e5449b3a507 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303313f7eb0825229442a8e787e5449b3a507 2023-03-31 16:56:20,389 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=325, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/A/da88ecdf5df8437f9cfa1f5ead0c9631 2023-03-31 16:56:20,398 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=325 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/B/9bde24cd7f5f4177ac731f146292efa2 2023-03-31 16:56:20,413 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=325 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/C/2ed2ba619a2045a8985ee92ac6f6f816 2023-03-31 16:56:20,417 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/A/da88ecdf5df8437f9cfa1f5ead0c9631 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/da88ecdf5df8437f9cfa1f5ead0c9631 2023-03-31 16:56:20,420 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/da88ecdf5df8437f9cfa1f5ead0c9631, entries=150, sequenceid=325, filesize=25.4 K 2023-03-31 16:56:20,421 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/B/9bde24cd7f5f4177ac731f146292efa2 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/9bde24cd7f5f4177ac731f146292efa2 2023-03-31 16:56:20,425 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/9bde24cd7f5f4177ac731f146292efa2, entries=150, sequenceid=325, filesize=11.9 K 2023-03-31 16:56:20,425 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/C/2ed2ba619a2045a8985ee92ac6f6f816 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/2ed2ba619a2045a8985ee92ac6f6f816 2023-03-31 16:56:20,428 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/2ed2ba619a2045a8985ee92ac6f6f816, entries=150, sequenceid=325, filesize=11.9 K 2023-03-31 16:56:20,429 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=0 B/0 for 313b3df11d64a9396698e7102a97394f in 63ms, sequenceid=325, compaction requested=true 2023-03-31 16:56:20,429 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 313b3df11d64a9396698e7102a97394f: 2023-03-31 16:56:20,429 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:20,429 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:20,429 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:20,429 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,429 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:20,429 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:20,431 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,431 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:20,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:20,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:20,431 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:20,431 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:20,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:20,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:20,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:20,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:20,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:20,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3457a552[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:20,433 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:20,433 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:20,434 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:20,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:20,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:20,434 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,434 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:20,434 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:20,434 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:20,434 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:20,436 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,436 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:20,436 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:20,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:20,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:20,437 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:20,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:20,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:20,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:20,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:20,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:20,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:20,439 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:20,439 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:20,439 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:20,439 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:20,439 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:20,439 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:20,441 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:20,441 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:20,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:20,441 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:20,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:20,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:20,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:20,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:20,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:20,442 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:20,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:20,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:20,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:20,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:20,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:20,443 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:20,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:20,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,445 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,445 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:20,445 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:20,445 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:20,446 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:20,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:20,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:20,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:20,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:20,446 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,446 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:20,446 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:20,446 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:20,446 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:20,446 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:56:20,446 DEBUG [Thread-4140] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:20,446 DEBUG [Thread-4140] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:20,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:20,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:20,546 DEBUG [Thread-4140] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:20,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:20,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:20,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:20,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:20,562 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:20,562 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:20,562 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:20,562 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:20,562 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:20,562 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:20,567 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:20,567 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:20,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:20,567 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:20,567 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:20,568 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:20,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:20,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:20,568 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:20,568 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:20,568 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:20,568 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:20,568 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:20,568 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:20,568 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:20,569 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:20,569 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. started... 2023-03-31 16:56:20,569 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 313b3df11d64a9396698e7102a97394f: 2023-03-31 16:56:20,569 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:20,569 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:20,569 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:20,569 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,569 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:20,569 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:20,572 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,572 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:20,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:20,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:20,573 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:20,573 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:20,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:20,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:20,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:20,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:20,574 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:20,574 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3effa8ea[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:20,574 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:20,574 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:20,576 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:20,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:20,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:20,576 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:20,576 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:20,576 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:20,576 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,576 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:20,577 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,577 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:20,577 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:20,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:20,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:20,577 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:20,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:20,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:20,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:20,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:20,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:20,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:20,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:20,580 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:20,580 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:20,580 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:20,580 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:20,580 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:20,582 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:20,582 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:20,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:20,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:20,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:20,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:20,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:20,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:20,582 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:20,583 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:20,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:20,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:20,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:20,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:20,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:20,584 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:20,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:20,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,587 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,587 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:20,587 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:20,587 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:20,587 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:56:20,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:20,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:20,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:20,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:20,587 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:20,587 DEBUG [Thread-4140] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:20,587 DEBUG [Thread-4140] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:20,587 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,587 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:20,587 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:20,587 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:20,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:20,587 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:20,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:20,687 DEBUG [Thread-4140] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:20,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:20,690 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:20,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:20,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:20,691 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:20,691 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:20,691 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:20,691 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:20,692 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:20,692 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:20,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on 313b3df11d64a9396698e7102a97394f 2023-03-31 16:56:20,693 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,693 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 313b3df11d64a9396698e7102a97394f 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-31 16:56:20,693 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:20,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:20,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:20,694 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,694 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:20,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:20,694 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=A 2023-03-31 16:56:20,694 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:56:20,694 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=B 2023-03-31 16:56:20,695 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:56:20,695 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:20,695 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=C 2023-03-31 16:56:20,695 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:56:20,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:20,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:20,695 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:20,695 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:20,695 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:20,696 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:20,696 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:20,696 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:20,696 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:20,696 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:20,696 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:20,696 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. started... 2023-03-31 16:56:20,696 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. as already flushing 2023-03-31 16:56:20,696 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:20,696 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:20,696 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:20,696 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,696 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:20,696 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:20,698 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:20,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:20,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:20,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:20,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:20,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:20,700 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:20,702 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:20,702 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:20,702 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:20,702 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c093c0[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:20,702 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:20,703 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:20,704 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:20,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:20,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:20,704 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:20,704 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:20,704 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:20,704 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,704 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:20,707 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:20,707 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:20,707 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,707 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:20,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:20,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:20,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:20,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:20,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:20,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:20,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:20,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:20,710 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:20,710 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:20,710 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:20,710 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:20,710 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:20,710 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:20,711 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:20,711 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:20,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:20,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:20,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:20,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:20,711 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:20,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:20,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:20,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:20,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:20,712 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:20,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:20,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:20,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:20,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,714 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:20,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:20,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,717 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,717 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:20,717 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:20,717 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:20,717 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:20,717 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,717 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:20,717 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:20,717 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:20,717 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:20,717 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:20,717 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:20,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:20,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:20,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:20,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:20,718 DEBUG [Thread-4140] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:20,718 DEBUG [Thread-4140] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:20,725 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:20,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49566 deadline: 1680281840720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,725 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:20,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49608 deadline: 1680281840721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,726 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1680281780717, max timestamp=1680281780723Num uniques 859; 2023-03-31 16:56:20,726 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 16:56:20,727 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1680281780717, max timestamp=1680281780723Num uniques 859; 2023-03-31 16:56:20,727 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 16:56:20,729 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680281780717, max timestamp=1680281780723Num uniques 859; 2023-03-31 16:56:20,729 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-03-31 16:56:20,737 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230331e7f3708c696b4bb39dc2afa84b1b16ce to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331e7f3708c696b4bb39dc2afa84b1b16ce 2023-03-31 16:56:20,754 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:20,754 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:20,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49602 deadline: 1680281840750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49618 deadline: 1680281840751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,755 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:20,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49568 deadline: 1680281840752, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,758 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=337, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/A/981fe047bd364ef384ac88e3d465f587 2023-03-31 16:56:20,778 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=337 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/B/63e9d6bf909a439f871cbffc37a5e9fb 2023-03-31 16:56:20,789 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=337 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/C/fdae65420483445f92bc922c07c09ffa 2023-03-31 16:56:20,794 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/A/981fe047bd364ef384ac88e3d465f587 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/981fe047bd364ef384ac88e3d465f587 2023-03-31 16:56:20,796 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/981fe047bd364ef384ac88e3d465f587, entries=400, sequenceid=337, filesize=59.8 K 2023-03-31 16:56:20,797 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/B/63e9d6bf909a439f871cbffc37a5e9fb as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/63e9d6bf909a439f871cbffc37a5e9fb 2023-03-31 16:56:20,801 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/63e9d6bf909a439f871cbffc37a5e9fb, entries=100, sequenceid=337, filesize=9.5 K 2023-03-31 16:56:20,802 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/C/fdae65420483445f92bc922c07c09ffa as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/fdae65420483445f92bc922c07c09ffa 2023-03-31 16:56:20,806 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/fdae65420483445f92bc922c07c09ffa, entries=100, sequenceid=337, filesize=9.5 K 2023-03-31 16:56:20,807 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=124.84 KB/127836 for 313b3df11d64a9396698e7102a97394f in 114ms, sequenceid=337, compaction requested=true 2023-03-31 16:56:20,807 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 313b3df11d64a9396698e7102a97394f: 2023-03-31 16:56:20,807 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:56:20,807 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:56:20,807 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-31 16:56:20,807 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-31 16:56:20,807 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:56:20,809 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 58882 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-31 16:56:20,810 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): 313b3df11d64a9396698e7102a97394f/B is initiating minor compaction (all files) 2023-03-31 16:56:20,810 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 313b3df11d64a9396698e7102a97394f/B in TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:20,810 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/5f1ff6b263554c819345d606c6ed8122, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/afe5223c9fc442118374543638e2af09, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/beff2d13f7ad418e8cd8a51bfc0b119b, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/9bde24cd7f5f4177ac731f146292efa2, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/63e9d6bf909a439f871cbffc37a5e9fb] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp, totalSize=57.5 K 2023-03-31 16:56:20,810 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 165830 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-31 16:56:20,810 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1912): 313b3df11d64a9396698e7102a97394f/A is initiating minor compaction (all files) 2023-03-31 16:56:20,810 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 313b3df11d64a9396698e7102a97394f/A in TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:20,810 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/04ce8f4061424fa487158be331c9e02d, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/dd1d4fe0b0b64966be099cb76c7aded0, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/5573c6fa554946628ea04eeaa68e9da2, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/da88ecdf5df8437f9cfa1f5ead0c9631, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/981fe047bd364ef384ac88e3d465f587] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp, totalSize=161.9 K 2023-03-31 16:56:20,811 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 04ce8f4061424fa487158be331c9e02d, keycount=150, bloomtype=ROW, size=25.8 K, encoding=NONE, compression=NONE, seqNum=241, earliestPutTs=1680281778232 2023-03-31 16:56:20,811 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 5f1ff6b263554c819345d606c6ed8122, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=241, earliestPutTs=1680281778232 2023-03-31 16:56:20,811 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting dd1d4fe0b0b64966be099cb76c7aded0, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=278, earliestPutTs=1680281778279 2023-03-31 16:56:20,811 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting afe5223c9fc442118374543638e2af09, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=278, earliestPutTs=1680281778279 2023-03-31 16:56:20,811 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 5573c6fa554946628ea04eeaa68e9da2, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=290, earliestPutTs=1680281778901 2023-03-31 16:56:20,812 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting beff2d13f7ad418e8cd8a51bfc0b119b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=290, earliestPutTs=1680281778901 2023-03-31 16:56:20,812 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting da88ecdf5df8437f9cfa1f5ead0c9631, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=325, earliestPutTs=1680281779546 2023-03-31 16:56:20,812 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 981fe047bd364ef384ac88e3d465f587, keycount=400, bloomtype=ROW, size=59.8 K, encoding=NONE, compression=NONE, seqNum=337, earliestPutTs=1680281780680 2023-03-31 16:56:20,812 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 9bde24cd7f5f4177ac731f146292efa2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=325, earliestPutTs=1680281779546 2023-03-31 16:56:20,813 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 63e9d6bf909a439f871cbffc37a5e9fb, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=337, earliestPutTs=1680281780692 2023-03-31 16:56:20,818 DEBUG [Thread-4140] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:20,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:20,820 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:20,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:20,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:20,822 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:20,822 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:20,822 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:20,822 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:20,823 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:20,823 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:20,828 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:20,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:20,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:20,828 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,829 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:20,829 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:20,829 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:20,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:20,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:20,830 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:20,832 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:20,832 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:20,832 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:20,832 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:20,832 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:20,834 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): 313b3df11d64a9396698e7102a97394f#B#compaction#624 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:56:20,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on 313b3df11d64a9396698e7102a97394f 2023-03-31 16:56:20,835 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 313b3df11d64a9396698e7102a97394f 3/3 column families, dataSize=131.55 KB heapSize=374.30 KB 2023-03-31 16:56:20,836 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=A 2023-03-31 16:56:20,836 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:56:20,836 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=B 2023-03-31 16:56:20,836 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:56:20,836 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=C 2023-03-31 16:56:20,836 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:56:20,840 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:20,841 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:20,841 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:20,841 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. started... 2023-03-31 16:56:20,841 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. as already flushing 2023-03-31 16:56:20,841 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:20,841 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:20,841 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:20,841 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,841 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:20,841 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:20,842 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:20,842 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:20,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:20,843 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] throttle.PressureAwareThroughputController(145): 313b3df11d64a9396698e7102a97394f#A#compaction#623 average throughput is 0.99 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:56:20,843 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:20,843 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:20,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:20,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:20,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:20,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:20,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:20,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f8bb0f7[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:20,849 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:20,849 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:20,851 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:20,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:20,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:20,851 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:20,851 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:20,851 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:20,851 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,851 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:20,853 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:20,853 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:20,853 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:20,853 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:20,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:20,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:20,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:20,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:20,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:20,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:20,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:20,856 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:20,856 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:20,856 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:20,856 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:20,856 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:20,856 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:20,859 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:20,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49566 deadline: 1680281840855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,859 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:20,859 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:20,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:20,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:20,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:20,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:20,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:20,859 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:20,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:20,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:20,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:20,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:20,860 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:20,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:20,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:20,861 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:20,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49602 deadline: 1680281840856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,862 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:20,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49568 deadline: 1680281840856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,862 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:20,862 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49618 deadline: 1680281840856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:20,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:20,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,864 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:20,864 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:20,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:20,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:20,865 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:20,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49608 deadline: 1680281840859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,865 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,865 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:20,865 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:20,865 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,865 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:20,865 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:20,865 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:20,865 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:56:20,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:20,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:20,866 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:20,866 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:20,866 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:20,866 DEBUG [Thread-4140] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:20,866 DEBUG [Thread-4140] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:20,870 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/B/9aa80338984346b8832d4517ef6bca0a as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/9aa80338984346b8832d4517ef6bca0a 2023-03-31 16:56:20,881 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 313b3df11d64a9396698e7102a97394f/B of 313b3df11d64a9396698e7102a97394f into 9aa80338984346b8832d4517ef6bca0a(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:56:20,882 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 313b3df11d64a9396698e7102a97394f: 2023-03-31 16:56:20,882 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f., storeName=313b3df11d64a9396698e7102a97394f/B, priority=11, startTime=1680281780807; duration=0sec 2023-03-31 16:56:20,882 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:56:20,882 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-31 16:56:20,883 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 58882 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-31 16:56:20,884 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): 313b3df11d64a9396698e7102a97394f/C is initiating minor compaction (all files) 2023-03-31 16:56:20,884 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 313b3df11d64a9396698e7102a97394f/C in TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:20,884 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/fcf6fe37d91e4ac49edfeb7d1761ee77, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/9ca45eb906334f76b38464b9659bc024, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/9b743e44bca748e19b9090ccd77cacbb, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/2ed2ba619a2045a8985ee92ac6f6f816, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/fdae65420483445f92bc922c07c09ffa] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp, totalSize=57.5 K 2023-03-31 16:56:20,884 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting fcf6fe37d91e4ac49edfeb7d1761ee77, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=241, earliestPutTs=1680281778232 2023-03-31 16:56:20,885 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 9ca45eb906334f76b38464b9659bc024, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=278, earliestPutTs=1680281778279 2023-03-31 16:56:20,885 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 9b743e44bca748e19b9090ccd77cacbb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=290, earliestPutTs=1680281778901 2023-03-31 16:56:20,885 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 2ed2ba619a2045a8985ee92ac6f6f816, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=325, earliestPutTs=1680281779546 2023-03-31 16:56:20,885 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting fdae65420483445f92bc922c07c09ffa, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=337, earliestPutTs=1680281780692 2023-03-31 16:56:20,892 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): 313b3df11d64a9396698e7102a97394f#C#compaction#626 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:56:20,893 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023033142808fd488d944aa9d66f761e64d259c to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033142808fd488d944aa9d66f761e64d259c 2023-03-31 16:56:20,903 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=374, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/A/89cd51876936497d86c5513148efa8ec 2023-03-31 16:56:20,905 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/C/915e0411902b4a238f2fc3675d32367b as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/915e0411902b4a238f2fc3675d32367b 2023-03-31 16:56:20,909 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 313b3df11d64a9396698e7102a97394f/C of 313b3df11d64a9396698e7102a97394f into 915e0411902b4a238f2fc3675d32367b(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:56:20,909 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 313b3df11d64a9396698e7102a97394f: 2023-03-31 16:56:20,910 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f., storeName=313b3df11d64a9396698e7102a97394f/C, priority=11, startTime=1680281780807; duration=0sec 2023-03-31 16:56:20,910 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:56:20,911 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=374 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/B/bad83eefb17748ffa963902d4d3644d8 2023-03-31 16:56:20,963 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:20,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49566 deadline: 1680281840960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,966 DEBUG [Thread-4140] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:20,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:20,967 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:20,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49608 deadline: 1680281840966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,968 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:20,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:20,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:20,969 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:20,969 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:20,969 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:20,969 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:20,970 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:20,970 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:20,971 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,971 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:20,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:20,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:20,971 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,971 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:20,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:20,972 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:20,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:20,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:20,972 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:20,972 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:20,973 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:20,973 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:20,973 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:20,973 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:20,973 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:20,973 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:20,973 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:20,973 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. started... 2023-03-31 16:56:20,973 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. as already flushing 2023-03-31 16:56:20,973 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:20,973 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:20,973 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:20,973 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,973 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:20,973 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:20,975 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,975 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:20,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:20,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:20,975 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:20,975 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:20,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:20,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:20,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:20,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:20,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:20,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@567ff361[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:20,976 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:20,976 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:20,979 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:20,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:20,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:20,979 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:20,979 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:20,979 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:20,979 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,979 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:20,980 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:20,980 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:20,980 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,980 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:20,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:20,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:20,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:20,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:20,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:20,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:20,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:20,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:20,982 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:20,982 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:20,983 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:20,983 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:20,983 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:20,983 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:20,984 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:20,984 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:20,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:20,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:20,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:20,984 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:20,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:20,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:20,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:20,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:20,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:20,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:20,985 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:20,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:20,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:20,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,986 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:20,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:20,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,989 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,989 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:20,989 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:20,989 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:20,989 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:20,989 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:20,989 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:20,989 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:20,989 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:20,989 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:20,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:20,989 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:20,989 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:20,989 DEBUG [Thread-4140] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:20,989 DEBUG [Thread-4140] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:20,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:20,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:20,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:21,064 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:21,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49602 deadline: 1680281841062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:21,064 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:21,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49568 deadline: 1680281841062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:21,065 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:21,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49618 deadline: 1680281841063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:21,089 DEBUG [Thread-4140] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:21,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:21,091 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:21,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:21,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:21,093 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:21,093 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:21,093 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:21,093 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:21,093 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:21,093 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:21,095 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:21,095 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:21,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:21,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:21,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:21,096 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:21,096 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:21,096 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:21,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:21,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:21,096 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:21,096 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:21,097 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:21,097 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:21,097 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:21,097 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:21,097 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:21,097 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:21,097 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:21,097 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. started... 2023-03-31 16:56:21,097 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. as already flushing 2023-03-31 16:56:21,097 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:21,097 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:21,097 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:21,097 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:21,097 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:21,097 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:21,099 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:21,099 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:21,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:21,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:21,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:21,099 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:21,099 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:21,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:21,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:21,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:21,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:21,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:21,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:21,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1584a9bc[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:21,101 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:21,101 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:21,102 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:21,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:21,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:21,103 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:21,103 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:21,103 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:21,103 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:21,103 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:21,104 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:21,104 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:21,104 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:21,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:21,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:21,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:21,104 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:21,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:21,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:21,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:21,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:21,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:21,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:21,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:21,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:21,107 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:21,107 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:21,107 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:21,107 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:21,107 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:21,107 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:21,109 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:21,109 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:21,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:21,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:21,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:21,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:21,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:21,109 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:21,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:21,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:21,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:21,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:21,110 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:21,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:21,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:21,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:21,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:21,111 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:21,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:21,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:21,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:21,114 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:21,114 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:21,114 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:21,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:21,114 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:21,114 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:21,114 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:21,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:21,114 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:21,114 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:21,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:21,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:21,114 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:21,115 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:21,115 DEBUG [Thread-4140] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:21,115 DEBUG [Thread-4140] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:21,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:21,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:21,166 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:21,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49566 deadline: 1680281841164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:21,173 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:21,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49608 deadline: 1680281841170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:21,215 DEBUG [Thread-4140] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:21,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:21,217 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:21,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:21,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:21,218 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:21,218 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:21,218 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:21,218 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:21,219 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:21,219 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:21,220 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:21,220 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:21,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:21,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:21,220 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:21,220 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:21,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:21,221 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:21,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:21,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:21,221 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:21,221 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:21,221 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:21,221 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:21,221 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:21,221 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:21,222 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:21,222 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:21,222 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:21,222 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. started... 2023-03-31 16:56:21,222 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. as already flushing 2023-03-31 16:56:21,222 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:21,222 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:21,222 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:21,222 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:21,222 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:21,222 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:21,224 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:21,224 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:21,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:21,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:21,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:21,224 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:21,224 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:21,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:21,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:21,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:21,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:21,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:21,226 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:21,226 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11de765f[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:21,227 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:21,227 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:21,231 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:21,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:21,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:21,231 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:21,231 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:21,232 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:21,231 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:21,232 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:21,233 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:21,233 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:21,233 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:21,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:21,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:21,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:21,233 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:21,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:21,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:21,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:21,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:21,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:21,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:21,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:21,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:21,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:21,236 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:21,237 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:21,237 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:21,237 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:21,237 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:21,238 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:21,239 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:21,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:21,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:21,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:21,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:21,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:21,239 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:21,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:21,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:21,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:21,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:21,240 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:21,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:21,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:21,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:21,241 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:21,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:21,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:21,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:21,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:21,243 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:21,243 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:21,243 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:21,243 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:21,243 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:21,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:21,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:21,243 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:21,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:21,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:21,243 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:21,243 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:21,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:21,244 DEBUG [Thread-4140] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:21,244 DEBUG [Thread-4140] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:21,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:21,244 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:21,244 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:21,244 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:56:21,318 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.33 KB at sequenceid=374 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/C/20feb655ad96480b858e0481fb790904 2023-03-31 16:56:21,322 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/A/89cd51876936497d86c5513148efa8ec as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/89cd51876936497d86c5513148efa8ec 2023-03-31 16:56:21,325 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/89cd51876936497d86c5513148efa8ec, entries=250, sequenceid=374, filesize=39.2 K 2023-03-31 16:56:21,326 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/B/bad83eefb17748ffa963902d4d3644d8 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/bad83eefb17748ffa963902d4d3644d8 2023-03-31 16:56:21,329 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/bad83eefb17748ffa963902d4d3644d8, entries=150, sequenceid=374, filesize=11.9 K 2023-03-31 16:56:21,330 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/C/20feb655ad96480b858e0481fb790904 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/20feb655ad96480b858e0481fb790904 2023-03-31 16:56:21,333 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/20feb655ad96480b858e0481fb790904, entries=150, sequenceid=374, filesize=11.9 K 2023-03-31 16:56:21,334 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~138.26 KB/141576, heapSize ~394.18 KB/403640, currentSize=40.25 KB/41220 for 313b3df11d64a9396698e7102a97394f in 499ms, sequenceid=374, compaction requested=false 2023-03-31 16:56:21,334 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 313b3df11d64a9396698e7102a97394f: 2023-03-31 16:56:21,344 DEBUG [Thread-4140] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:21,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:21,346 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:21,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:21,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:21,347 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:21,347 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:21,347 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:21,347 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:21,347 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:21,347 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:21,349 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:21,349 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:21,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:21,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:21,350 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:21,350 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:21,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:21,350 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:21,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:21,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:21,351 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:21,351 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:21,351 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:21,351 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:21,351 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:21,351 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:21,351 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:21,351 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:21,352 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:21,352 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. started... 2023-03-31 16:56:21,352 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 313b3df11d64a9396698e7102a97394f 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-03-31 16:56:21,352 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=A 2023-03-31 16:56:21,352 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:56:21,352 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=B 2023-03-31 16:56:21,352 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:56:21,352 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=C 2023-03-31 16:56:21,352 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:56:21,366 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230331d550ba2a1f764528855d8f9815bdab11 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331d550ba2a1f764528855d8f9815bdab11 2023-03-31 16:56:21,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on 313b3df11d64a9396698e7102a97394f 2023-03-31 16:56:21,367 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. as already flushing 2023-03-31 16:56:21,371 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=385, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/A/cd8f70664250474e93489da2ecfb266e 2023-03-31 16:56:21,390 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680281781385, max timestamp=1680281781389Num uniques 859; 2023-03-31 16:56:21,390 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 16:56:21,391 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680281781385, max timestamp=1680281781389Num uniques 859; 2023-03-31 16:56:21,391 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 16:56:21,391 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680281781385, max timestamp=1680281781389Num uniques 859; 2023-03-31 16:56:21,391 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 16:56:21,421 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680281781402, max timestamp=1680281781418Num uniques 1018; 2023-03-31 16:56:21,421 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680281781402, max timestamp=1680281781418Num uniques 1018; 2023-03-31 16:56:21,421 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680281781402, max timestamp=1680281781418Num uniques 1018; 2023-03-31 16:56:21,421 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-03-31 16:56:21,421 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-03-31 16:56:21,421 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-03-31 16:56:21,444 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:21,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49568 deadline: 1680281841440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:21,446 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:21,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49602 deadline: 1680281841442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:21,446 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:21,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49618 deadline: 1680281841443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:21,472 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:21,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49566 deadline: 1680281841469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:21,477 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:21,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49608 deadline: 1680281841475, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:21,548 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:21,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49568 deadline: 1680281841545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:21,549 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:21,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49602 deadline: 1680281841547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:21,549 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:21,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49618 deadline: 1680281841547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:21,706 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/A/3ac01c86f9f94a6a8731c8c1ddd5c599 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/3ac01c86f9f94a6a8731c8c1ddd5c599 2023-03-31 16:56:21,711 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 313b3df11d64a9396698e7102a97394f/A of 313b3df11d64a9396698e7102a97394f into 3ac01c86f9f94a6a8731c8c1ddd5c599(size=26.1 K), total size for store is 65.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:56:21,711 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 313b3df11d64a9396698e7102a97394f: 2023-03-31 16:56:21,711 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f., storeName=313b3df11d64a9396698e7102a97394f/A, priority=11, startTime=1680281780807; duration=0sec 2023-03-31 16:56:21,711 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:56:21,752 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:21,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49568 deadline: 1680281841750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:21,753 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:21,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49602 deadline: 1680281841750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:21,753 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:21,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49618 deadline: 1680281841751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:21,781 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=385 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/B/9a8f221abe2b4022a984ad62526e0b8f 2023-03-31 16:56:21,791 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=385 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/C/3bbd80c48d7b4a91a7c50c82965ba696 2023-03-31 16:56:21,794 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/A/cd8f70664250474e93489da2ecfb266e as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/cd8f70664250474e93489da2ecfb266e 2023-03-31 16:56:21,798 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/cd8f70664250474e93489da2ecfb266e, entries=150, sequenceid=385, filesize=25.4 K 2023-03-31 16:56:21,799 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/B/9a8f221abe2b4022a984ad62526e0b8f as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/9a8f221abe2b4022a984ad62526e0b8f 2023-03-31 16:56:21,802 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/9a8f221abe2b4022a984ad62526e0b8f, entries=150, sequenceid=385, filesize=11.9 K 2023-03-31 16:56:21,803 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/C/3bbd80c48d7b4a91a7c50c82965ba696 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/3bbd80c48d7b4a91a7c50c82965ba696 2023-03-31 16:56:21,806 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/3bbd80c48d7b4a91a7c50c82965ba696, entries=150, sequenceid=385, filesize=11.9 K 2023-03-31 16:56:21,807 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 313b3df11d64a9396698e7102a97394f in 455ms, sequenceid=385, compaction requested=true 2023-03-31 16:56:21,807 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 313b3df11d64a9396698e7102a97394f: 2023-03-31 16:56:21,807 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:21,807 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:21,807 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:21,807 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:21,807 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:21,807 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:21,809 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:21,809 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:21,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:21,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:21,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:21,809 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:21,809 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:21,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:21,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:21,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:21,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:21,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:21,810 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:21,810 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13d0609b[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:21,810 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:21,811 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:21,812 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:21,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:21,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:21,812 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:21,812 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:21,812 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:21,812 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:21,812 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:21,813 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:21,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:21,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:21,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:21,814 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:21,814 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:21,814 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:21,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:21,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:21,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:21,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:21,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:21,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:21,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:21,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:21,816 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:21,816 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:21,816 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:21,816 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:21,816 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:21,816 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:21,820 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:21,820 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:21,820 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:21,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:21,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:21,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:21,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:21,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:21,821 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:21,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:21,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:21,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:21,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:21,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:21,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:21,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:21,822 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:21,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:21,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:21,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:21,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:21,824 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:21,824 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:21,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:21,824 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:21,824 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:21,824 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:21,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:21,824 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:21,824 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:21,824 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:21,824 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:21,824 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:21,824 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:21,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:21,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:21,824 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:56:21,825 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:21,824 DEBUG [Thread-4140] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:21,825 DEBUG [Thread-4140] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:21,925 DEBUG [Thread-4140] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:21,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:21,927 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:21,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:21,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:21,928 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:21,928 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:21,928 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:21,928 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:21,929 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:21,929 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:21,930 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:21,930 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:21,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:21,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:21,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:21,931 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:21,931 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:21,931 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:21,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:21,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:21,931 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:21,932 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:21,932 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:21,932 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:21,932 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:21,932 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:21,932 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:21,932 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:21,932 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:21,932 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. started... 2023-03-31 16:56:21,932 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 313b3df11d64a9396698e7102a97394f 3/3 column families, dataSize=140.89 KB heapSize=394.69 KB 2023-03-31 16:56:21,932 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=A 2023-03-31 16:56:21,933 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:56:21,933 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=B 2023-03-31 16:56:21,933 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:56:21,933 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=C 2023-03-31 16:56:21,933 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:56:21,945 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023033163e9cc7123544744b4014236d602e63e to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033163e9cc7123544744b4014236d602e63e 2023-03-31 16:56:21,954 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=439, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/A/76731a4c2ed14a328e1255343dd68c90 2023-03-31 16:56:21,963 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=439 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/B/5a7fa7b61d0945dbbd90c01655fba9b2 2023-03-31 16:56:21,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on 313b3df11d64a9396698e7102a97394f 2023-03-31 16:56:21,979 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. as already flushing 2023-03-31 16:56:21,989 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:21,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49608 deadline: 1680281841986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:21,990 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:21,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49566 deadline: 1680281841989, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,056 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:22,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49602 deadline: 1680281842054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,056 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:22,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49568 deadline: 1680281842054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,056 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:22,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49618 deadline: 1680281842055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,092 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:22,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49608 deadline: 1680281842090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,093 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:22,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49566 deadline: 1680281842091, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,295 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:22,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49566 deadline: 1680281842294, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,296 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:22,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49608 deadline: 1680281842294, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,373 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=439 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/C/c3928f2ef8794f908165b2f1b50dffb1 2023-03-31 16:56:22,376 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/A/76731a4c2ed14a328e1255343dd68c90 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/76731a4c2ed14a328e1255343dd68c90 2023-03-31 16:56:22,380 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/76731a4c2ed14a328e1255343dd68c90, entries=150, sequenceid=439, filesize=25.4 K 2023-03-31 16:56:22,380 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/B/5a7fa7b61d0945dbbd90c01655fba9b2 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/5a7fa7b61d0945dbbd90c01655fba9b2 2023-03-31 16:56:22,395 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/5a7fa7b61d0945dbbd90c01655fba9b2, entries=150, sequenceid=439, filesize=11.9 K 2023-03-31 16:56:22,396 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/C/c3928f2ef8794f908165b2f1b50dffb1 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/c3928f2ef8794f908165b2f1b50dffb1 2023-03-31 16:56:22,400 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/c3928f2ef8794f908165b2f1b50dffb1, entries=150, sequenceid=439, filesize=11.9 K 2023-03-31 16:56:22,400 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~394.64 KB/404112, currentSize=40.25 KB/41220 for 313b3df11d64a9396698e7102a97394f in 468ms, sequenceid=439, compaction requested=true 2023-03-31 16:56:22,400 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 313b3df11d64a9396698e7102a97394f: 2023-03-31 16:56:22,400 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:22,400 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:22,400 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:22,400 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,404 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:22,404 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:22,406 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:22,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:22,406 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:22,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:22,407 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:22,407 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:22,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:22,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:22,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:22,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:22,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a1d6d7b[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:22,408 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:22,408 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:22,410 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:22,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:22,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:22,410 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:22,410 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:22,410 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:22,410 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,410 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:22,411 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,411 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:22,411 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:22,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:22,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:22,411 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:22,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:22,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:22,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:22,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:22,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:22,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:22,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:22,414 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:22,414 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:22,414 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:22,414 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:22,414 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:22,416 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:22,416 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:22,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:22,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:22,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:22,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:22,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:22,416 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:22,416 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:22,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:22,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:22,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:22,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:22,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:22,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:22,417 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:22,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:22,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,419 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:22,420 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:22,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:22,420 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:22,420 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:22,420 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:22,420 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:22,420 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:22,420 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:22,420 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,420 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:22,420 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:22,420 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:22,420 DEBUG [Thread-4140] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:22,420 DEBUG [Thread-4140] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:22,420 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:56:22,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:22,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:22,520 DEBUG [Thread-4140] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:22,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:22,522 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:22,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:22,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:22,523 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:22,523 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:22,523 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:22,523 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:22,524 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:22,524 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:22,526 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,526 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:22,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:22,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:22,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:22,526 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,526 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:22,527 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:22,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:22,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:22,527 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:22,527 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:22,527 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:22,527 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:22,527 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:22,527 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:22,528 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:22,528 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:22,528 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:22,528 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. started... 2023-03-31 16:56:22,528 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 313b3df11d64a9396698e7102a97394f 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-03-31 16:56:22,528 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=A 2023-03-31 16:56:22,528 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:56:22,528 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=B 2023-03-31 16:56:22,528 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:56:22,528 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=C 2023-03-31 16:56:22,528 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:56:22,539 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230331589fa342861545e8b23693b4c8d931b7 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331589fa342861545e8b23693b4c8d931b7 2023-03-31 16:56:22,544 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=448, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/A/07c0dcfbd8384c0ea809fb8aebf5b3f4 2023-03-31 16:56:22,552 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=448 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/B/925e506e414b49d693cb635b7be2e8e8 2023-03-31 16:56:22,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on 313b3df11d64a9396698e7102a97394f 2023-03-31 16:56:22,559 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. as already flushing 2023-03-31 16:56:22,562 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=448 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/C/9480f34e9a3141c4acfb9dd1fea570a1 2023-03-31 16:56:22,565 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/A/07c0dcfbd8384c0ea809fb8aebf5b3f4 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/07c0dcfbd8384c0ea809fb8aebf5b3f4 2023-03-31 16:56:22,573 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/07c0dcfbd8384c0ea809fb8aebf5b3f4, entries=100, sequenceid=448, filesize=18.6 K 2023-03-31 16:56:22,574 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/B/925e506e414b49d693cb635b7be2e8e8 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/925e506e414b49d693cb635b7be2e8e8 2023-03-31 16:56:22,577 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/925e506e414b49d693cb635b7be2e8e8, entries=100, sequenceid=448, filesize=9.5 K 2023-03-31 16:56:22,577 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/C/9480f34e9a3141c4acfb9dd1fea570a1 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/9480f34e9a3141c4acfb9dd1fea570a1 2023-03-31 16:56:22,581 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/9480f34e9a3141c4acfb9dd1fea570a1, entries=100, sequenceid=448, filesize=9.5 K 2023-03-31 16:56:22,581 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=73.80 KB/75570 for 313b3df11d64a9396698e7102a97394f in 53ms, sequenceid=448, compaction requested=true 2023-03-31 16:56:22,581 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 313b3df11d64a9396698e7102a97394f: 2023-03-31 16:56:22,581 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:22,581 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:22,582 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:22,582 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,582 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:22,582 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:22,583 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,583 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:22,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:22,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:22,583 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:22,583 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:22,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:22,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:22,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on 313b3df11d64a9396698e7102a97394f 2023-03-31 16:56:22,584 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 313b3df11d64a9396698e7102a97394f 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-31 16:56:22,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:22,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,585 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=A 2023-03-31 16:56:22,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:22,585 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:56:22,585 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=B 2023-03-31 16:56:22,585 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:56:22,585 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=C 2023-03-31 16:56:22,585 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:56:22,585 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:22,585 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a64facd[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:22,585 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:22,586 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:22,587 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:22,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:22,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:22,587 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,587 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:22,587 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:22,587 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:22,587 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:22,592 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:22,592 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:22,592 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:22,592 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:22,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:22,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:22,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:22,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:22,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:22,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:22,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:22,595 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:22,595 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:22,595 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:22,595 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:22,595 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:22,595 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:22,597 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:22,597 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:22,597 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:22,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:22,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:22,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:22,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:22,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:22,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:22,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:22,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:22,600 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:22,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:22,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:22,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:22,601 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:22,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:22,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,604 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:22,604 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:22,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:22,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:22,605 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,605 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:22,605 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:22,605 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,605 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:22,605 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:22,605 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:22,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:22,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:22,605 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:22,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:22,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:22,605 DEBUG [Thread-4140] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:22,605 DEBUG [Thread-4140] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:22,606 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:22,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49618 deadline: 1680281842602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,606 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:56:22,608 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303310eb8f78746064daf855da8567d296539 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303310eb8f78746064daf855da8567d296539 2023-03-31 16:56:22,608 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:22,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49602 deadline: 1680281842603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,609 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:22,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49566 deadline: 1680281842603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,609 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:22,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49608 deadline: 1680281842604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,610 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:22,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49568 deadline: 1680281842606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,613 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=465, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/A/d7aba367d4e94d7f83d53fc6d760d754 2023-03-31 16:56:22,625 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=465 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/B/e85d7be9433d42f785db9a9c0433ae6c 2023-03-31 16:56:22,640 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=465 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/C/a3956fabe99b4d429b02a4b8c8634937 2023-03-31 16:56:22,643 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/A/d7aba367d4e94d7f83d53fc6d760d754 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/d7aba367d4e94d7f83d53fc6d760d754 2023-03-31 16:56:22,646 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/d7aba367d4e94d7f83d53fc6d760d754, entries=300, sequenceid=465, filesize=46.1 K 2023-03-31 16:56:22,647 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/B/e85d7be9433d42f785db9a9c0433ae6c as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/e85d7be9433d42f785db9a9c0433ae6c 2023-03-31 16:56:22,650 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/e85d7be9433d42f785db9a9c0433ae6c, entries=150, sequenceid=465, filesize=11.9 K 2023-03-31 16:56:22,651 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/C/a3956fabe99b4d429b02a4b8c8634937 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/a3956fabe99b4d429b02a4b8c8634937 2023-03-31 16:56:22,653 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/a3956fabe99b4d429b02a4b8c8634937, entries=150, sequenceid=465, filesize=11.9 K 2023-03-31 16:56:22,654 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 313b3df11d64a9396698e7102a97394f in 70ms, sequenceid=465, compaction requested=true 2023-03-31 16:56:22,654 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 313b3df11d64a9396698e7102a97394f: 2023-03-31 16:56:22,654 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:56:22,654 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:56:22,654 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:56:22,654 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-31 16:56:22,654 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-31 16:56:22,656 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 185152 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-31 16:56:22,656 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71388 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-31 16:56:22,656 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1912): 313b3df11d64a9396698e7102a97394f/A is initiating minor compaction (all files) 2023-03-31 16:56:22,656 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): 313b3df11d64a9396698e7102a97394f/B is initiating minor compaction (all files) 2023-03-31 16:56:22,656 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 313b3df11d64a9396698e7102a97394f/A in TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:22,656 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 313b3df11d64a9396698e7102a97394f/B in TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:22,656 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/3ac01c86f9f94a6a8731c8c1ddd5c599, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/89cd51876936497d86c5513148efa8ec, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/cd8f70664250474e93489da2ecfb266e, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/76731a4c2ed14a328e1255343dd68c90, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/07c0dcfbd8384c0ea809fb8aebf5b3f4, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/d7aba367d4e94d7f83d53fc6d760d754] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp, totalSize=180.8 K 2023-03-31 16:56:22,656 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/9aa80338984346b8832d4517ef6bca0a, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/bad83eefb17748ffa963902d4d3644d8, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/9a8f221abe2b4022a984ad62526e0b8f, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/5a7fa7b61d0945dbbd90c01655fba9b2, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/925e506e414b49d693cb635b7be2e8e8, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/e85d7be9433d42f785db9a9c0433ae6c] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp, totalSize=69.7 K 2023-03-31 16:56:22,657 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 3ac01c86f9f94a6a8731c8c1ddd5c599, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=337, earliestPutTs=1680281779546 2023-03-31 16:56:22,657 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 9aa80338984346b8832d4517ef6bca0a, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=337, earliestPutTs=1680281779546 2023-03-31 16:56:22,657 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 89cd51876936497d86c5513148efa8ec, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=374, earliestPutTs=1680281780736 2023-03-31 16:56:22,657 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting bad83eefb17748ffa963902d4d3644d8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=374, earliestPutTs=1680281780736 2023-03-31 16:56:22,657 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting cd8f70664250474e93489da2ecfb266e, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=385, earliestPutTs=1680281780848 2023-03-31 16:56:22,657 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 9a8f221abe2b4022a984ad62526e0b8f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=385, earliestPutTs=1680281780848 2023-03-31 16:56:22,657 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 76731a4c2ed14a328e1255343dd68c90, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=439, earliestPutTs=1680281781436 2023-03-31 16:56:22,658 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 5a7fa7b61d0945dbbd90c01655fba9b2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=439, earliestPutTs=1680281781436 2023-03-31 16:56:22,658 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 07c0dcfbd8384c0ea809fb8aebf5b3f4, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=448, earliestPutTs=1680281781985 2023-03-31 16:56:22,658 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 925e506e414b49d693cb635b7be2e8e8, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=448, earliestPutTs=1680281781985 2023-03-31 16:56:22,658 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting d7aba367d4e94d7f83d53fc6d760d754, keycount=300, bloomtype=ROW, size=46.1 K, encoding=NONE, compression=NONE, seqNum=465, earliestPutTs=1680281782577 2023-03-31 16:56:22,658 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting e85d7be9433d42f785db9a9c0433ae6c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=465, earliestPutTs=1680281782578 2023-03-31 16:56:22,669 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): 313b3df11d64a9396698e7102a97394f#B#compaction#642 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:56:22,674 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] throttle.PressureAwareThroughputController(145): 313b3df11d64a9396698e7102a97394f#A#compaction#641 average throughput is 2.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:56:22,684 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/B/2713f94bf7104e62828c8de3f3c1546e as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/2713f94bf7104e62828c8de3f3c1546e 2023-03-31 16:56:22,688 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 313b3df11d64a9396698e7102a97394f/B of 313b3df11d64a9396698e7102a97394f into 2713f94bf7104e62828c8de3f3c1546e(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:56:22,688 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 313b3df11d64a9396698e7102a97394f: 2023-03-31 16:56:22,688 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f., storeName=313b3df11d64a9396698e7102a97394f/B, priority=10, startTime=1680281782654; duration=0sec 2023-03-31 16:56:22,688 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:56:22,688 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-31 16:56:22,690 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71388 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-31 16:56:22,690 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): 313b3df11d64a9396698e7102a97394f/C is initiating minor compaction (all files) 2023-03-31 16:56:22,690 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 313b3df11d64a9396698e7102a97394f/C in TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:22,690 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/915e0411902b4a238f2fc3675d32367b, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/20feb655ad96480b858e0481fb790904, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/3bbd80c48d7b4a91a7c50c82965ba696, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/c3928f2ef8794f908165b2f1b50dffb1, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/9480f34e9a3141c4acfb9dd1fea570a1, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/a3956fabe99b4d429b02a4b8c8634937] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp, totalSize=69.7 K 2023-03-31 16:56:22,690 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 915e0411902b4a238f2fc3675d32367b, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=337, earliestPutTs=1680281779546 2023-03-31 16:56:22,691 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 20feb655ad96480b858e0481fb790904, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=374, earliestPutTs=1680281780736 2023-03-31 16:56:22,692 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 3bbd80c48d7b4a91a7c50c82965ba696, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=385, earliestPutTs=1680281780848 2023-03-31 16:56:22,692 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting c3928f2ef8794f908165b2f1b50dffb1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=439, earliestPutTs=1680281781436 2023-03-31 16:56:22,692 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 9480f34e9a3141c4acfb9dd1fea570a1, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=448, earliestPutTs=1680281781985 2023-03-31 16:56:22,693 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting a3956fabe99b4d429b02a4b8c8634937, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=465, earliestPutTs=1680281782578 2023-03-31 16:56:22,703 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): 313b3df11d64a9396698e7102a97394f#C#compaction#643 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:56:22,703 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/A/8c9817cf36734a5eb2019812038e2ee6 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/8c9817cf36734a5eb2019812038e2ee6 2023-03-31 16:56:22,705 DEBUG [Thread-4140] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:22,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:22,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:22,709 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 313b3df11d64a9396698e7102a97394f/A of 313b3df11d64a9396698e7102a97394f into 8c9817cf36734a5eb2019812038e2ee6(size=26.3 K), total size for store is 26.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:56:22,709 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 313b3df11d64a9396698e7102a97394f: 2023-03-31 16:56:22,709 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f., storeName=313b3df11d64a9396698e7102a97394f/A, priority=10, startTime=1680281782654; duration=0sec 2023-03-31 16:56:22,709 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:56:22,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:22,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:22,711 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:22,711 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:22,711 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:22,711 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:22,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on 313b3df11d64a9396698e7102a97394f 2023-03-31 16:56:22,711 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:22,711 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 313b3df11d64a9396698e7102a97394f 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-31 16:56:22,711 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:22,712 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=A 2023-03-31 16:56:22,712 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:56:22,712 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=B 2023-03-31 16:56:22,712 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:56:22,712 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=C 2023-03-31 16:56:22,712 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:56:22,713 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,713 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:22,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:22,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:22,714 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/C/34cf8bf76a654dcab805588a759cc936 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/34cf8bf76a654dcab805588a759cc936 2023-03-31 16:56:22,718 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,718 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:22,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:22,718 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:22,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:22,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:22,719 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:22,719 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:22,719 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:22,719 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:22,719 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:22,719 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:22,719 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:22,719 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:22,719 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:22,719 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. started... 2023-03-31 16:56:22,719 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. as already flushing 2023-03-31 16:56:22,719 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:22,720 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:22,720 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:22,720 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,720 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 313b3df11d64a9396698e7102a97394f/C of 313b3df11d64a9396698e7102a97394f into 34cf8bf76a654dcab805588a759cc936(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:56:22,720 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:22,720 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 313b3df11d64a9396698e7102a97394f: 2023-03-31 16:56:22,720 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:22,720 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f., storeName=313b3df11d64a9396698e7102a97394f/C, priority=10, startTime=1680281782654; duration=0sec 2023-03-31 16:56:22,720 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:56:22,722 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:22,722 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:22,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:22,722 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:22,723 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:22,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:22,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:22,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:22,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:22,725 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:22,725 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@686a1950[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:22,725 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:22,725 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:22,727 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:22,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:22,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:22,727 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:22,727 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:22,727 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303313609ae91b80249cebf4ba29f4e95a6a2 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303313609ae91b80249cebf4ba29f4e95a6a2 2023-03-31 16:56:22,727 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:22,728 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:22,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49618 deadline: 1680281842725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,729 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:22,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49568 deadline: 1680281842725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,729 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:22,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49566 deadline: 1680281842726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,729 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:22,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49608 deadline: 1680281842726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,729 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:22,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49602 deadline: 1680281842727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,730 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:22,730 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:22,730 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:22,736 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,736 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:22,737 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:22,737 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:22,737 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:22,737 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:22,737 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:22,737 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:22,739 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:22,739 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:22,739 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:22,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:22,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:22,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:22,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:22,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:22,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:22,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:22,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:22,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:22,740 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:22,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:22,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:22,742 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:22,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:22,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,746 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:22,746 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:22,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:22,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:22,746 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:22,746 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:22,746 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:22,746 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,746 DEBUG [Thread-4140] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:22,747 DEBUG [Thread-4140] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:22,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:22,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:22,747 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:22,747 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:22,747 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,747 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:22,747 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:22,747 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:22,746 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:56:22,831 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:22,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49618 deadline: 1680281842829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,831 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:22,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49568 deadline: 1680281842830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,832 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:22,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49566 deadline: 1680281842830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,832 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:22,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49608 deadline: 1680281842830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,832 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:22,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49602 deadline: 1680281842830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,847 DEBUG [Thread-4140] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:22,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:22,849 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:22,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:22,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:22,850 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:22,850 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:22,850 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:22,850 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:22,850 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:22,851 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:22,852 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:22,852 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:22,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:22,852 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,852 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:22,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:22,853 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:22,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:22,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:22,853 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:22,853 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:22,853 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:22,853 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:22,853 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:22,853 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:22,854 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:22,854 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:22,854 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:22,854 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. started... 2023-03-31 16:56:22,854 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. as already flushing 2023-03-31 16:56:22,854 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:22,854 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:22,855 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:22,855 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,855 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:22,855 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:22,856 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:22,856 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:22,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:22,856 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:22,856 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:22,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:22,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:22,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:22,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:22,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:22,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78ae787e[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:22,858 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:22,858 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:22,859 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:22,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:22,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:22,859 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:22,859 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:22,859 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,859 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:22,859 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:22,862 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:22,862 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,862 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:22,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:22,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:22,862 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:22,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:22,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:22,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:22,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:22,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:22,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:22,864 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:22,864 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:22,864 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:22,864 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:22,864 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:22,864 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:22,869 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:22,869 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:22,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:22,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:22,869 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:22,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:22,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:22,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:22,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:22,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:22,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:22,869 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:22,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:22,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:22,870 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:22,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:22,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:22,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,873 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,873 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:22,873 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:22,873 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:22,873 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,873 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:22,873 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:22,873 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:22,874 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:56:22,873 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:22,874 DEBUG [Thread-4140] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:22,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:22,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:22,873 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:22,874 DEBUG [Thread-4140] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:22,874 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:22,874 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:22,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:22,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:22,974 DEBUG [Thread-4140] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:22,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:22,976 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:22,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:22,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:22,977 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:22,978 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:22,978 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:22,978 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:22,978 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:22,978 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:22,979 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:22,979 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:22,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:22,980 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,980 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:22,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:22,980 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:22,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:22,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:22,981 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:22,982 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:22,982 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:22,982 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:22,982 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:22,982 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:22,982 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:22,982 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:22,982 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:22,982 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. started... 2023-03-31 16:56:22,982 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. as already flushing 2023-03-31 16:56:22,982 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:22,982 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:22,982 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:22,983 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,983 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:22,983 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:22,985 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:22,985 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:22,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:22,986 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:22,986 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:22,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:22,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:22,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:22,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:22,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:22,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2989b4a5[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:22,987 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:22,987 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:22,989 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:22,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:22,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:22,989 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:22,989 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:22,989 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:22,989 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,990 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:22,991 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:22,991 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:22,991 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:22,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:22,991 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:22,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:22,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:22,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:22,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:22,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:22,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:22,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:22,994 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:22,994 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:22,994 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:22,994 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:22,994 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:22,995 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:22,995 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:22,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:22,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:22,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:22,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:22,995 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:22,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:22,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:22,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:22,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:22,996 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:22,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:22,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:22,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:22,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,997 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:22,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:22,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:22,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:23,003 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:23,003 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:23,004 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:56:23,004 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:23,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:23,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:23,004 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:23,004 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:23,004 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:23,004 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:23,004 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:23,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:23,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:23,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:23,004 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:23,004 DEBUG [Thread-4140] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:23,004 DEBUG [Thread-4140] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:23,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:23,004 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:23,034 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:23,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49568 deadline: 1680281843032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:23,034 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:23,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49608 deadline: 1680281843033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:23,034 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:23,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49602 deadline: 1680281843033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:23,035 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:23,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49618 deadline: 1680281843033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:23,035 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:23,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49566 deadline: 1680281843033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:23,105 DEBUG [Thread-4140] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:23,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:23,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:23,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:23,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:23,108 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:23,108 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:23,108 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:23,108 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:23,108 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:23,108 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:23,109 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:23,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:23,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:23,109 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:23,110 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:23,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:23,110 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:23,110 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:23,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:23,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:23,110 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:23,111 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:23,111 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:23,111 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:23,111 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:23,111 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:23,111 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:23,111 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:23,111 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:23,111 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. started... 2023-03-31 16:56:23,111 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. as already flushing 2023-03-31 16:56:23,111 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:23,111 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:23,111 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:23,111 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:23,111 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:23,111 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:23,113 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:23,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:23,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:23,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:23,113 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:23,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:23,114 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:23,114 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:23,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:23,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:23,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:23,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:23,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:23,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41691cb[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:23,115 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:23,115 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:23,117 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:23,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:23,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:23,117 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:23,117 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:23,117 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:23,117 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:23,117 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:23,119 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:23,119 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:23,119 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:23,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:23,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:23,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:23,119 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:23,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:23,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:23,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:23,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:23,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:23,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:23,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:23,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:23,122 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:23,122 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:23,123 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:23,123 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:23,123 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:23,123 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:23,124 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:23,124 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:23,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:23,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:23,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:23,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:23,124 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:23,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:23,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:23,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:23,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:23,125 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:23,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:23,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:23,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:23,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:23,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:23,127 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:23,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:23,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:23,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:23,129 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:23,129 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:23,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:23,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:23,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:23,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:23,130 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:23,130 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:23,130 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:23,130 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:23,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:23,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:23,130 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:23,130 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:23,130 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:23,130 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:23,130 DEBUG [Thread-4140] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:23,130 DEBUG [Thread-4140] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:23,146 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=484, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/A/d342afa65b3943b5bd542ce5ad6be095 2023-03-31 16:56:23,156 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=484 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/B/49a73bf8bd8e435dabc74cfa8581a782 2023-03-31 16:56:23,167 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=484 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/C/2d5e1c6bee084c3193507c9821cb2ec2 2023-03-31 16:56:23,171 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/A/d342afa65b3943b5bd542ce5ad6be095 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/d342afa65b3943b5bd542ce5ad6be095 2023-03-31 16:56:23,174 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/d342afa65b3943b5bd542ce5ad6be095, entries=250, sequenceid=484, filesize=39.2 K 2023-03-31 16:56:23,175 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/B/49a73bf8bd8e435dabc74cfa8581a782 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/49a73bf8bd8e435dabc74cfa8581a782 2023-03-31 16:56:23,178 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/49a73bf8bd8e435dabc74cfa8581a782, entries=150, sequenceid=484, filesize=11.9 K 2023-03-31 16:56:23,179 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/C/2d5e1c6bee084c3193507c9821cb2ec2 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/2d5e1c6bee084c3193507c9821cb2ec2 2023-03-31 16:56:23,181 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/2d5e1c6bee084c3193507c9821cb2ec2, entries=150, sequenceid=484, filesize=11.9 K 2023-03-31 16:56:23,182 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 313b3df11d64a9396698e7102a97394f in 471ms, sequenceid=484, compaction requested=false 2023-03-31 16:56:23,182 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 313b3df11d64a9396698e7102a97394f: 2023-03-31 16:56:23,230 DEBUG [Thread-4140] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:23,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:23,233 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:23,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:23,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:23,234 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:23,234 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:23,234 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:23,234 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:23,235 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:23,235 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:23,236 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:23,236 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:23,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:23,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:23,237 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:23,237 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:23,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:23,237 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:23,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:23,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:23,237 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:23,238 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:23,238 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:23,238 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:23,238 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:23,238 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:23,238 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:23,238 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:23,238 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:23,238 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. started... 2023-03-31 16:56:23,238 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 313b3df11d64a9396698e7102a97394f 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-31 16:56:23,239 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=A 2023-03-31 16:56:23,239 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:56:23,239 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=B 2023-03-31 16:56:23,239 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:56:23,239 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=C 2023-03-31 16:56:23,239 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:56:23,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on 313b3df11d64a9396698e7102a97394f 2023-03-31 16:56:23,338 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. as already flushing 2023-03-31 16:56:23,349 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:23,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49566 deadline: 1680281843346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:23,350 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:23,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49602 deadline: 1680281843347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:23,350 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:23,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49608 deadline: 1680281843347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:23,352 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:23,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49618 deadline: 1680281843349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:23,353 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:23,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49568 deadline: 1680281843350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:23,452 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:23,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49608 deadline: 1680281843451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:23,453 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:23,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49602 deadline: 1680281843451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:23,453 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:23,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49566 deadline: 1680281843451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:23,456 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:23,457 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:23,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49618 deadline: 1680281843453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:23,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49568 deadline: 1680281843453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:23,657 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:23,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49608 deadline: 1680281843654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:23,657 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:23,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49602 deadline: 1680281843655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:23,658 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:23,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49566 deadline: 1680281843655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:23,661 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:23,661 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303317555983aa23247349385e7a44a6a921f to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303317555983aa23247349385e7a44a6a921f 2023-03-31 16:56:23,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49618 deadline: 1680281843658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:23,662 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:23,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49568 deadline: 1680281843659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:23,961 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:23,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49608 deadline: 1680281843959, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:23,961 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:23,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49602 deadline: 1680281843960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:23,961 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:23,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49566 deadline: 1680281843960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:23,966 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:23,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49618 deadline: 1680281843964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:23,966 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:23,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49568 deadline: 1680281843964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:24,066 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=500, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/A/4a764573fca647129013af99f06ef144 2023-03-31 16:56:24,463 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:24,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49608 deadline: 1680281844462, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:24,466 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:24,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49602 deadline: 1680281844464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:24,467 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:24,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49566 deadline: 1680281844466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:24,470 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:24,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49568 deadline: 1680281844468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:24,470 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:24,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49618 deadline: 1680281844468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:24,476 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=500 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/B/28d2f2cc47a242b8b64f29a9a663b794 2023-03-31 16:56:24,484 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=500 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/C/8c21ee7f2d1d42c99a6f0495e40b65ce 2023-03-31 16:56:24,488 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/A/4a764573fca647129013af99f06ef144 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/4a764573fca647129013af99f06ef144 2023-03-31 16:56:24,491 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/4a764573fca647129013af99f06ef144, entries=150, sequenceid=500, filesize=25.4 K 2023-03-31 16:56:24,491 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/B/28d2f2cc47a242b8b64f29a9a663b794 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/28d2f2cc47a242b8b64f29a9a663b794 2023-03-31 16:56:24,495 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/28d2f2cc47a242b8b64f29a9a663b794, entries=150, sequenceid=500, filesize=11.9 K 2023-03-31 16:56:24,496 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/C/8c21ee7f2d1d42c99a6f0495e40b65ce as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/8c21ee7f2d1d42c99a6f0495e40b65ce 2023-03-31 16:56:24,499 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/8c21ee7f2d1d42c99a6f0495e40b65ce, entries=150, sequenceid=500, filesize=11.9 K 2023-03-31 16:56:24,499 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=100.63 KB/103050 for 313b3df11d64a9396698e7102a97394f in 1261ms, sequenceid=500, compaction requested=true 2023-03-31 16:56:24,499 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 313b3df11d64a9396698e7102a97394f: 2023-03-31 16:56:24,499 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:24,499 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:24,499 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:24,500 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:24,500 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:24,500 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:24,501 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:24,501 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:24,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:24,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:24,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:24,501 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:24,501 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:24,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:24,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:24,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:24,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:24,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:24,503 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:24,503 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c8ea880[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:24,503 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:24,503 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:24,504 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:24,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:24,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:24,504 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:24,504 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:24,504 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:24,504 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:24,504 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:24,506 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:24,506 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:24,506 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:24,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:24,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:24,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:24,506 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:24,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:24,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:24,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:24,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:24,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:24,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:24,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:24,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:24,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:24,509 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:24,509 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:24,509 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:24,509 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:24,509 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:24,516 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:24,516 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:24,517 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:24,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:24,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:24,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:24,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:24,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:24,517 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:24,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:24,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:24,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:24,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:24,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:24,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:24,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:24,519 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:24,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:24,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:24,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:24,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:24,521 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:24,521 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:24,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:24,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:24,521 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:24,521 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:24,521 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:24,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:24,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:24,521 DEBUG [Thread-4140] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:24,521 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:24,521 DEBUG [Thread-4140] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:24,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:24,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:24,521 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:24,521 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:24,521 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:24,521 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:24,522 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:56:24,621 DEBUG [Thread-4140] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:24,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:24,623 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:24,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:24,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:24,625 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:24,625 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:24,625 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:24,625 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:24,625 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:24,625 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:24,626 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:24,626 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:24,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:24,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:24,627 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:24,627 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:24,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:24,627 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:24,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:24,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:24,627 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:24,628 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:24,628 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:24,628 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:24,628 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:24,628 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:24,628 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:24,628 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:24,628 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:24,628 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. started... 2023-03-31 16:56:24,628 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 313b3df11d64a9396698e7102a97394f 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-31 16:56:24,629 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=A 2023-03-31 16:56:24,629 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:56:24,629 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=B 2023-03-31 16:56:24,629 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:56:24,629 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=C 2023-03-31 16:56:24,629 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:56:24,640 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303313a1dec17291c423d97381321f86594b2 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303313a1dec17291c423d97381321f86594b2 2023-03-31 16:56:25,046 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=518, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/A/202b0e2bcd154b65ac8a2459537deee1 2023-03-31 16:56:25,058 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=518 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/B/cbe85d56c0584bb198e635a9c3715ba1 2023-03-31 16:56:25,068 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=518 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/C/4cfccbbe58cb42c8897cde42382679b4 2023-03-31 16:56:25,071 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/A/202b0e2bcd154b65ac8a2459537deee1 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/202b0e2bcd154b65ac8a2459537deee1 2023-03-31 16:56:25,075 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/202b0e2bcd154b65ac8a2459537deee1, entries=150, sequenceid=518, filesize=25.4 K 2023-03-31 16:56:25,075 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/B/cbe85d56c0584bb198e635a9c3715ba1 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/cbe85d56c0584bb198e635a9c3715ba1 2023-03-31 16:56:25,078 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/cbe85d56c0584bb198e635a9c3715ba1, entries=150, sequenceid=518, filesize=11.9 K 2023-03-31 16:56:25,079 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/C/4cfccbbe58cb42c8897cde42382679b4 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/4cfccbbe58cb42c8897cde42382679b4 2023-03-31 16:56:25,082 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/4cfccbbe58cb42c8897cde42382679b4, entries=150, sequenceid=518, filesize=11.9 K 2023-03-31 16:56:25,083 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=0 B/0 for 313b3df11d64a9396698e7102a97394f in 455ms, sequenceid=518, compaction requested=true 2023-03-31 16:56:25,083 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 313b3df11d64a9396698e7102a97394f: 2023-03-31 16:56:25,083 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:25,083 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:25,083 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:25,083 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,083 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:25,083 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:25,085 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,085 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:25,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:25,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:25,085 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:25,085 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:25,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:25,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:25,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:25,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:25,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:25,087 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37e84ae7[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:25,087 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:25,087 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:25,088 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:25,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:25,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:25,089 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:25,089 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:25,089 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,089 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:25,089 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:25,090 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,090 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:25,090 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:25,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:25,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:25,090 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:25,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:25,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:25,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:25,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:25,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:25,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:25,093 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:25,093 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:25,093 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:25,093 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:25,093 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:25,093 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:25,095 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:25,095 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:25,095 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:25,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:25,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:25,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:25,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:25,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:25,095 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:25,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:25,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:25,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:25,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:25,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:25,097 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:25,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:25,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:25,103 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:25,103 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:25,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:25,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:25,103 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:25,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:25,103 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:25,104 DEBUG [Thread-4140] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:25,104 DEBUG [Thread-4140] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:25,103 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:25,104 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:56:25,104 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:25,104 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,104 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:25,104 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:25,104 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:25,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:25,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:25,204 DEBUG [Thread-4140] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:25,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:25,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:25,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:25,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:25,212 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:25,212 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:25,212 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:25,212 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:25,213 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:25,213 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:25,214 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:25,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:25,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:25,214 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:25,215 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,215 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:25,215 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:25,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:25,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:25,215 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:25,215 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:25,216 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:25,216 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:25,216 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:25,216 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:25,220 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:25,221 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:25,221 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:25,222 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. started... 2023-03-31 16:56:25,222 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 313b3df11d64a9396698e7102a97394f: 2023-03-31 16:56:25,222 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:25,222 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:25,222 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:25,222 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,222 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:25,222 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:25,224 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,224 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:25,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:25,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:25,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:25,225 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:25,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:25,225 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:25,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:25,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:25,226 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:25,226 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53a0fc23[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:25,226 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:25,226 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:25,227 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:25,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:25,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:25,227 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,227 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:25,227 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:25,227 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:25,227 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:25,230 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,230 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:25,230 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:25,231 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:25,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:25,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:25,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:25,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:25,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:25,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:25,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:25,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:25,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:25,233 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:25,233 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:25,233 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:25,233 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:25,233 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:25,235 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:25,235 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:25,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:25,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:25,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:25,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:25,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:25,235 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:25,236 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:25,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:25,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:25,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:25,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:25,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:25,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:25,237 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:25,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:25,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,239 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,240 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:25,240 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:25,240 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,240 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:25,240 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:25,240 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:25,240 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:25,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:25,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:25,240 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:25,240 DEBUG [Thread-4140] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:25,240 DEBUG [Thread-4140] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:25,240 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:25,240 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:25,240 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:25,240 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:56:25,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:25,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:25,340 DEBUG [Thread-4140] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:25,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:25,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:25,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:25,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:25,344 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:25,344 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:25,344 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:25,344 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:25,344 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:25,344 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:25,346 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:25,346 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:25,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:25,346 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:25,346 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:25,346 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:25,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:25,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:25,347 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:25,347 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:25,347 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:25,347 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:25,347 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:25,347 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:25,347 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:25,347 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:25,347 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:25,347 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. started... 2023-03-31 16:56:25,348 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 313b3df11d64a9396698e7102a97394f: 2023-03-31 16:56:25,348 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:25,348 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:25,348 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:25,348 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,348 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:25,348 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:25,350 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:25,350 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:25,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:25,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:25,351 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:25,351 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:25,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:25,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:25,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:25,352 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:25,352 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6388a755[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:25,352 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:25,352 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:25,353 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:25,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:25,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:25,354 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:25,354 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:25,354 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:25,354 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,354 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:25,355 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,355 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:25,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:25,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:25,356 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:25,356 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:25,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:25,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:25,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:25,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:25,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:25,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:25,358 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:25,358 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:25,358 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:25,358 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:25,359 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:25,359 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:25,361 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:25,361 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:25,361 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:25,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:25,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:25,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:25,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:25,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:25,361 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:25,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:25,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:25,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:25,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:25,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:25,362 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:25,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:25,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:25,366 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:25,366 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:25,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:25,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:25,366 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,366 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:25,366 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:25,366 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:25,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:25,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:25,366 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:56:25,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:25,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:25,366 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,367 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:25,367 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:25,367 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:25,367 DEBUG [Thread-4140] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:25,367 DEBUG [Thread-4140] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:25,467 DEBUG [Thread-4140] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:25,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:25,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:25,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:25,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:25,470 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:25,471 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:25,471 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:25,471 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:25,471 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:25,471 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:25,472 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:25,473 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:25,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:25,473 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,473 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:25,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:25,473 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:25,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:25,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:25,474 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:25,474 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:25,474 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:25,474 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:25,474 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:25,474 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:25,474 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:25,474 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:25,474 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:25,474 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. started... 2023-03-31 16:56:25,474 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 313b3df11d64a9396698e7102a97394f: 2023-03-31 16:56:25,475 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:25,475 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:25,475 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:25,475 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,475 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:25,475 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:25,477 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,477 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:25,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:25,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:25,477 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:25,477 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:25,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:25,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:25,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:25,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:25,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:25,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a0bd90c[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:25,479 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:25,479 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:25,481 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:25,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:25,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:25,481 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:25,481 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:25,481 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:25,481 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,481 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:25,484 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:25,484 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:25,484 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,484 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:25,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:25,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:25,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:25,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:25,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:25,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:25,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:25,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:25,486 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:25,486 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:25,487 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:25,487 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:25,487 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:25,487 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:25,488 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:25,488 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:25,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:25,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:25,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:25,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:25,488 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:25,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:25,489 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:25,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:25,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:25,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:25,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:25,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:25,491 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:25,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:25,511 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:25,511 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:25,511 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:25,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:25,511 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:25,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:25,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:25,511 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:25,511 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,511 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:25,511 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:25,511 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:25,511 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:25,511 DEBUG [Thread-4140] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:25,511 DEBUG [Thread-4140] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:25,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:25,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:25,512 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:56:25,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on 313b3df11d64a9396698e7102a97394f 2023-03-31 16:56:25,519 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 313b3df11d64a9396698e7102a97394f 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-31 16:56:25,519 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=A 2023-03-31 16:56:25,519 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:56:25,520 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=B 2023-03-31 16:56:25,520 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:56:25,520 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=C 2023-03-31 16:56:25,520 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:56:25,536 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303312442210389ce480eb1e3928489a2a2c2 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303312442210389ce480eb1e3928489a2a2c2 2023-03-31 16:56:25,551 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680281785539, max timestamp=1680281785549Num uniques 859; 2023-03-31 16:56:25,551 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 16:56:25,552 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680281785539, max timestamp=1680281785549Num uniques 859; 2023-03-31 16:56:25,552 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 16:56:25,552 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680281785539, max timestamp=1680281785550Num uniques 859; 2023-03-31 16:56:25,552 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 16:56:25,558 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=532, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/A/0c573347799c4d70a899fdad86c1060a 2023-03-31 16:56:25,571 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:25,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49608 deadline: 1680281845565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,571 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:25,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49568 deadline: 1680281845566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,574 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:25,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49566 deadline: 1680281845567, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,574 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:25,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49602 deadline: 1680281845568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,575 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:25,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49618 deadline: 1680281845569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,612 DEBUG [Thread-4140] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:25,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:25,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:25,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:25,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:25,615 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:25,615 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:25,615 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:25,615 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:25,615 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:25,615 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:25,620 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:25,620 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:25,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:25,620 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,620 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:25,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:25,620 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:25,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:25,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:25,621 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:25,621 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:25,621 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:25,621 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:25,621 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:25,621 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:25,621 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:25,621 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:25,621 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. started... 2023-03-31 16:56:25,621 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. as already flushing 2023-03-31 16:56:25,621 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:25,621 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:25,621 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:25,622 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,622 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:25,622 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:25,623 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,623 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:25,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:25,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:25,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:25,623 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:25,624 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:25,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:25,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:25,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:25,625 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:25,625 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5fc101d8[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:25,625 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:25,625 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:25,626 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:25,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:25,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:25,626 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:25,626 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:25,626 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:25,626 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,626 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:25,627 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:25,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:25,627 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:25,627 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:25,627 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:25,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:25,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:25,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:25,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:25,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:25,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:25,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:25,630 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:25,630 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:25,630 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:25,630 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:25,630 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:25,632 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:25,632 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:25,632 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:25,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:25,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:25,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:25,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:25,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:25,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:25,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:25,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:25,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:25,633 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:25,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:25,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:25,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,635 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:25,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:25,637 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,637 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:25,637 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:25,637 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,637 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:25,637 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:25,637 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:25,637 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:25,637 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:25,637 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:25,637 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:25,637 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:25,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:25,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:25,637 DEBUG [Thread-4140] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:25,637 DEBUG [Thread-4140] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:25,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:25,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:25,637 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:56:25,675 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:25,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49608 deadline: 1680281845672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,675 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:25,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49568 deadline: 1680281845672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,677 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:25,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49566 deadline: 1680281845675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,677 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:25,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49618 deadline: 1680281845676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,677 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:25,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49602 deadline: 1680281845676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,737 DEBUG [Thread-4140] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:25,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:25,740 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:25,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:25,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:25,742 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:25,742 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:25,742 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:25,742 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:25,742 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:25,742 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:25,743 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,743 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:25,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:25,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:25,744 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,744 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:25,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:25,744 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:25,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:25,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:25,745 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:25,745 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:25,745 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:25,745 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:25,745 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:25,745 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:25,745 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:25,745 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:25,745 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:25,746 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. started... 2023-03-31 16:56:25,746 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. as already flushing 2023-03-31 16:56:25,746 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:25,746 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:25,746 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:25,746 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,746 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:25,746 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:25,747 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:25,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:25,748 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:25,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:25,748 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:25,748 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:25,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:25,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:25,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:25,749 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:25,749 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@bd4bfe7[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:25,749 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:25,749 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:25,752 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:25,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:25,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:25,752 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,752 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:25,752 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:25,752 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:25,752 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:25,753 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,753 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:25,753 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:25,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:25,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:25,753 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:25,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:25,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:25,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:25,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:25,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:25,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:25,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:25,757 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:25,757 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:25,757 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:25,757 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:25,757 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:25,759 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:25,759 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:25,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:25,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:25,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:25,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:25,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:25,759 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:25,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:25,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:25,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:25,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:25,760 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:25,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:25,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:25,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:25,762 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:25,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,765 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,765 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:25,765 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:25,765 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:25,765 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:25,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:25,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:25,765 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,765 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:25,765 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:25,766 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:25,765 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:25,765 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:25,766 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:25,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:25,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:25,766 DEBUG [Thread-4140] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:25,766 DEBUG [Thread-4140] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:25,866 DEBUG [Thread-4140] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:25,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:25,868 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:25,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:25,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:25,869 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:25,869 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:25,870 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:25,870 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:25,870 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:25,870 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:25,872 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:25,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:25,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:25,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:25,873 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:25,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:25,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:25,873 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:25,874 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:25,874 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:25,874 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:25,874 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:25,874 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:25,874 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:25,874 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:25,874 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:25,874 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. started... 2023-03-31 16:56:25,874 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. as already flushing 2023-03-31 16:56:25,874 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:25,874 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:25,874 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:25,874 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,874 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:25,874 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:25,876 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:25,876 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:25,876 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:25,878 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:25,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49568 deadline: 1680281845878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,879 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:25,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49618 deadline: 1680281845878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,879 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:25,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49602 deadline: 1680281845878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,880 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,880 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,881 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:25,881 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a08fcf8[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:25,881 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:25,881 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:25,881 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:25,881 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:25,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49608 deadline: 1680281845880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,881 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:25,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49566 deadline: 1680281845880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,882 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:25,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:25,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:25,882 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:25,882 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:25,882 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:25,883 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,883 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:25,884 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,884 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:25,884 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:25,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:25,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:25,884 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:25,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:25,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:25,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:25,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:25,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:25,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:25,887 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:25,887 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:25,887 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:25,887 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:25,887 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:25,887 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:25,890 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:25,890 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:25,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:25,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:25,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:25,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:25,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:25,890 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:25,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:25,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:25,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:25,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:25,892 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:25,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:25,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:25,893 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:25,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:25,895 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,895 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:25,895 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:25,895 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:25,896 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:25,896 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:25,896 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:25,896 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:25,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:25,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:25,896 DEBUG [Thread-4140] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:25,896 DEBUG [Thread-4140] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:25,895 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:25,896 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:25,896 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:25,896 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:25,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:25,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:25,897 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:56:25,976 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=532 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/B/d8d0a65ea0674c35bd4ffa6f3c4b4bc3 2023-03-31 16:56:25,985 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=532 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/C/98adfa9865b9481fb1da72363f3419cb 2023-03-31 16:56:25,989 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/A/0c573347799c4d70a899fdad86c1060a as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/0c573347799c4d70a899fdad86c1060a 2023-03-31 16:56:25,992 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/0c573347799c4d70a899fdad86c1060a, entries=400, sequenceid=532, filesize=59.8 K 2023-03-31 16:56:25,994 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/B/d8d0a65ea0674c35bd4ffa6f3c4b4bc3 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/d8d0a65ea0674c35bd4ffa6f3c4b4bc3 2023-03-31 16:56:25,996 DEBUG [Thread-4140] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:25,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:25,998 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/d8d0a65ea0674c35bd4ffa6f3c4b4bc3, entries=150, sequenceid=532, filesize=11.9 K 2023-03-31 16:56:25,998 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/C/98adfa9865b9481fb1da72363f3419cb as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/98adfa9865b9481fb1da72363f3419cb 2023-03-31 16:56:26,001 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/98adfa9865b9481fb1da72363f3419cb, entries=150, sequenceid=532, filesize=11.9 K 2023-03-31 16:56:26,002 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 313b3df11d64a9396698e7102a97394f in 483ms, sequenceid=532, compaction requested=true 2023-03-31 16:56:26,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:26,002 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 313b3df11d64a9396698e7102a97394f: 2023-03-31 16:56:26,002 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:56:26,002 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-31 16:56:26,002 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-31 16:56:26,002 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:56:26,003 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:56:26,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:26,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:26,004 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:26,004 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:26,004 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:26,004 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:26,005 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:26,005 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:26,005 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 180442 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-31 16:56:26,005 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): 313b3df11d64a9396698e7102a97394f/A is initiating minor compaction (all files) 2023-03-31 16:56:26,005 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 313b3df11d64a9396698e7102a97394f/A in TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:26,005 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/8c9817cf36734a5eb2019812038e2ee6, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/d342afa65b3943b5bd542ce5ad6be095, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/4a764573fca647129013af99f06ef144, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/202b0e2bcd154b65ac8a2459537deee1, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/0c573347799c4d70a899fdad86c1060a] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp, totalSize=176.2 K 2023-03-31 16:56:26,006 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 8c9817cf36734a5eb2019812038e2ee6, keycount=150, bloomtype=ROW, size=26.3 K, encoding=NONE, compression=NONE, seqNum=465, earliestPutTs=1680281782578 2023-03-31 16:56:26,006 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:26,006 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:26,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:26,006 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61850 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-31 16:56:26,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:26,006 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1912): 313b3df11d64a9396698e7102a97394f/B is initiating minor compaction (all files) 2023-03-31 16:56:26,006 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 313b3df11d64a9396698e7102a97394f/B in TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:26,007 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/2713f94bf7104e62828c8de3f3c1546e, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/49a73bf8bd8e435dabc74cfa8581a782, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/28d2f2cc47a242b8b64f29a9a663b794, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/cbe85d56c0584bb198e635a9c3715ba1, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/d8d0a65ea0674c35bd4ffa6f3c4b4bc3] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp, totalSize=60.4 K 2023-03-31 16:56:26,007 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:26,007 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:26,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:26,007 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:26,007 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting d342afa65b3943b5bd542ce5ad6be095, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=484, earliestPutTs=1680281782602 2023-03-31 16:56:26,007 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 2713f94bf7104e62828c8de3f3c1546e, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=465, earliestPutTs=1680281782578 2023-03-31 16:56:26,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:26,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:26,008 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:26,008 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 4a764573fca647129013af99f06ef144, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=500, earliestPutTs=1680281782723 2023-03-31 16:56:26,008 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 49a73bf8bd8e435dabc74cfa8581a782, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=484, earliestPutTs=1680281782605 2023-03-31 16:56:26,009 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 202b0e2bcd154b65ac8a2459537deee1, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=518, earliestPutTs=1680281783345 2023-03-31 16:56:26,009 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 28d2f2cc47a242b8b64f29a9a663b794, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=500, earliestPutTs=1680281782723 2023-03-31 16:56:26,010 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting cbe85d56c0584bb198e635a9c3715ba1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=518, earliestPutTs=1680281783345 2023-03-31 16:56:26,010 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 0c573347799c4d70a899fdad86c1060a, keycount=400, bloomtype=ROW, size=59.8 K, encoding=NONE, compression=NONE, seqNum=532, earliestPutTs=1680281785490 2023-03-31 16:56:26,010 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting d8d0a65ea0674c35bd4ffa6f3c4b4bc3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=532, earliestPutTs=1680281785491 2023-03-31 16:56:26,017 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:26,017 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:26,017 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:26,017 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:26,017 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:26,017 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:26,017 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:26,017 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:26,017 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. started... 2023-03-31 16:56:26,018 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 313b3df11d64a9396698e7102a97394f 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-03-31 16:56:26,018 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=A 2023-03-31 16:56:26,018 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:56:26,018 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=B 2023-03-31 16:56:26,018 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:56:26,018 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=C 2023-03-31 16:56:26,018 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:56:26,032 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] throttle.PressureAwareThroughputController(145): 313b3df11d64a9396698e7102a97394f#B#compaction#657 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:56:26,033 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): 313b3df11d64a9396698e7102a97394f#A#compaction#656 average throughput is 1.52 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:56:26,061 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/B/f1fd3e829a03446e8ede1c55cbaec48a as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/f1fd3e829a03446e8ede1c55cbaec48a 2023-03-31 16:56:26,065 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 313b3df11d64a9396698e7102a97394f/B of 313b3df11d64a9396698e7102a97394f into f1fd3e829a03446e8ede1c55cbaec48a(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:56:26,065 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 313b3df11d64a9396698e7102a97394f: 2023-03-31 16:56:26,065 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f., storeName=313b3df11d64a9396698e7102a97394f/B, priority=11, startTime=1680281786002; duration=0sec 2023-03-31 16:56:26,065 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:56:26,065 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-31 16:56:26,066 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61850 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-31 16:56:26,066 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1912): 313b3df11d64a9396698e7102a97394f/C is initiating minor compaction (all files) 2023-03-31 16:56:26,067 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 313b3df11d64a9396698e7102a97394f/C in TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:26,067 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/34cf8bf76a654dcab805588a759cc936, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/2d5e1c6bee084c3193507c9821cb2ec2, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/8c21ee7f2d1d42c99a6f0495e40b65ce, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/4cfccbbe58cb42c8897cde42382679b4, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/98adfa9865b9481fb1da72363f3419cb] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp, totalSize=60.4 K 2023-03-31 16:56:26,067 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 34cf8bf76a654dcab805588a759cc936, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=465, earliestPutTs=1680281782578 2023-03-31 16:56:26,067 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 2d5e1c6bee084c3193507c9821cb2ec2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=484, earliestPutTs=1680281782605 2023-03-31 16:56:26,067 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 8c21ee7f2d1d42c99a6f0495e40b65ce, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=500, earliestPutTs=1680281782723 2023-03-31 16:56:26,068 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 4cfccbbe58cb42c8897cde42382679b4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=518, earliestPutTs=1680281783345 2023-03-31 16:56:26,068 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 98adfa9865b9481fb1da72363f3419cb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=532, earliestPutTs=1680281785491 2023-03-31 16:56:26,077 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] throttle.PressureAwareThroughputController(145): 313b3df11d64a9396698e7102a97394f#C#compaction#659 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:56:26,088 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/C/f691f015cb3a43c3bf17fd7ca7b13f0f as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/f691f015cb3a43c3bf17fd7ca7b13f0f 2023-03-31 16:56:26,091 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 313b3df11d64a9396698e7102a97394f/C of 313b3df11d64a9396698e7102a97394f into f691f015cb3a43c3bf17fd7ca7b13f0f(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:56:26,092 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 313b3df11d64a9396698e7102a97394f: 2023-03-31 16:56:26,092 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f., storeName=313b3df11d64a9396698e7102a97394f/C, priority=11, startTime=1680281786003; duration=0sec 2023-03-31 16:56:26,092 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:56:26,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on 313b3df11d64a9396698e7102a97394f 2023-03-31 16:56:26,181 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. as already flushing 2023-03-31 16:56:26,195 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:26,196 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:26,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49568 deadline: 1680281846190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:26,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49608 deadline: 1680281846190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:26,200 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:26,200 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:26,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49566 deadline: 1680281846195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:26,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49602 deadline: 1680281846196, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:26,201 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:26,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49618 deadline: 1680281846196, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:26,300 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:26,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49568 deadline: 1680281846297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:26,300 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:26,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49608 deadline: 1680281846297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:26,304 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:26,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49602 deadline: 1680281846302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:26,305 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:26,305 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:26,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49566 deadline: 1680281846302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:26,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49618 deadline: 1680281846302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:26,453 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230331aea0ee43858243deadbdd38525c61019 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331aea0ee43858243deadbdd38525c61019 2023-03-31 16:56:26,458 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=568, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/A/6dc56de547bb4e1b9b26a01c48ab1aae 2023-03-31 16:56:26,503 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:26,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49568 deadline: 1680281846501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:26,504 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:26,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49608 deadline: 1680281846501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:26,510 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:26,510 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:26,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49602 deadline: 1680281846505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:26,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49618 deadline: 1680281846506, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:26,510 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:26,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49566 deadline: 1680281846506, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:26,806 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:26,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49568 deadline: 1680281846804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:26,808 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:26,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49608 deadline: 1680281846806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:26,813 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:26,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49602 deadline: 1680281846812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:26,813 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:26,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49566 deadline: 1680281846812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:26,815 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:26,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49618 deadline: 1680281846813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:26,868 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=568 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/B/bfce5e92feec4a8ba92ed249fa70ec5f 2023-03-31 16:56:26,877 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=568 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/C/6dd9689ef20e41e08bc9f56a4c194358 2023-03-31 16:56:26,882 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/A/6dc56de547bb4e1b9b26a01c48ab1aae as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/6dc56de547bb4e1b9b26a01c48ab1aae 2023-03-31 16:56:26,885 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/6dc56de547bb4e1b9b26a01c48ab1aae, entries=150, sequenceid=568, filesize=25.4 K 2023-03-31 16:56:26,886 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/B/bfce5e92feec4a8ba92ed249fa70ec5f as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/bfce5e92feec4a8ba92ed249fa70ec5f 2023-03-31 16:56:26,889 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/bfce5e92feec4a8ba92ed249fa70ec5f, entries=150, sequenceid=568, filesize=11.9 K 2023-03-31 16:56:26,889 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/C/6dd9689ef20e41e08bc9f56a4c194358 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/6dd9689ef20e41e08bc9f56a4c194358 2023-03-31 16:56:26,894 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/6dd9689ef20e41e08bc9f56a4c194358, entries=150, sequenceid=568, filesize=11.9 K 2023-03-31 16:56:26,894 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for 313b3df11d64a9396698e7102a97394f in 876ms, sequenceid=568, compaction requested=false 2023-03-31 16:56:26,894 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 313b3df11d64a9396698e7102a97394f: 2023-03-31 16:56:26,895 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:26,895 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:26,895 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:26,895 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:26,896 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:26,896 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:26,901 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:26,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:26,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:26,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:26,901 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:26,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:26,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:26,902 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:26,902 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:26,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:26,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:26,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:26,902 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:26,902 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2dc013c7[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:26,902 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:26,903 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:26,904 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:26,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:26,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:26,904 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:26,904 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:26,904 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:26,904 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:26,904 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:26,906 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:26,906 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:26,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:26,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:26,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:26,906 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:26,906 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:26,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:26,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:26,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:26,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:26,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:26,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:26,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:26,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:26,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:26,908 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:26,908 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:26,908 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:26,908 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:26,909 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:26,911 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:26,911 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:26,911 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:26,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:26,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:26,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:26,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:26,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:26,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:26,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:26,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:26,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:26,912 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:26,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:26,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:26,913 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:26,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:26,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:26,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:26,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:26,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:26,916 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:26,916 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:26,916 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:26,916 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:26,917 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:26,917 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:26,917 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:26,917 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:26,917 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:26,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:26,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:26,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:26,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:26,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:26,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:26,917 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:26,917 DEBUG [Thread-4140] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:26,917 DEBUG [Thread-4140] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:27,018 DEBUG [Thread-4140] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:27,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:27,020 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:27,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:27,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:27,021 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:27,021 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:27,021 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:27,021 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:27,022 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:27,022 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:27,025 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:27,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:27,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:27,025 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:27,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:27,026 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:27,026 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:27,026 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:27,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:27,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:27,026 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:27,027 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:27,027 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:27,027 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:27,027 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:27,027 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:27,027 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:27,027 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:27,027 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:27,027 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. started... 2023-03-31 16:56:27,027 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 313b3df11d64a9396698e7102a97394f 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-31 16:56:27,028 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=A 2023-03-31 16:56:27,028 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:56:27,028 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=B 2023-03-31 16:56:27,028 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:56:27,028 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=C 2023-03-31 16:56:27,028 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:56:27,058 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303317bf20aa0ba424b2eb7115ac10432dd7f to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303317bf20aa0ba424b2eb7115ac10432dd7f 2023-03-31 16:56:27,256 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/A/700a7c97e96a4bff8157ca4bd033ce14 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/700a7c97e96a4bff8157ca4bd033ce14 2023-03-31 16:56:27,261 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 313b3df11d64a9396698e7102a97394f/A of 313b3df11d64a9396698e7102a97394f into 700a7c97e96a4bff8157ca4bd033ce14(size=26.5 K), total size for store is 51.9 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-03-31 16:56:27,261 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 313b3df11d64a9396698e7102a97394f: 2023-03-31 16:56:27,261 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f., storeName=313b3df11d64a9396698e7102a97394f/A, priority=11, startTime=1680281786002; duration=1sec 2023-03-31 16:56:27,262 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:56:27,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on 313b3df11d64a9396698e7102a97394f 2023-03-31 16:56:27,312 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. as already flushing 2023-03-31 16:56:27,328 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:27,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49608 deadline: 1680281847325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:27,328 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680281787322, max timestamp=1680281787326Num uniques 859; 2023-03-31 16:56:27,328 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:27,328 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 16:56:27,329 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680281787322, max timestamp=1680281787326Num uniques 859; 2023-03-31 16:56:27,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49618 deadline: 1680281847326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:27,329 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 16:56:27,329 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680281787322, max timestamp=1680281787326Num uniques 859; 2023-03-31 16:56:27,329 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 16:56:27,345 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:27,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49568 deadline: 1680281847342, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:27,346 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:27,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49566 deadline: 1680281847343, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:27,346 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:27,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49602 deadline: 1680281847343, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:27,432 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:27,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49618 deadline: 1680281847429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:27,432 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:27,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49608 deadline: 1680281847429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:27,449 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:27,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49568 deadline: 1680281847446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:27,449 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:27,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49566 deadline: 1680281847447, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:27,449 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:27,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49602 deadline: 1680281847447, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:27,464 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=581, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/A/08061d65a0a341e794ff5555c28719cf 2023-03-31 16:56:27,634 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:27,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49618 deadline: 1680281847633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:27,636 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:27,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49608 deadline: 1680281847634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:27,651 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:27,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49602 deadline: 1680281847650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:27,654 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:27,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49568 deadline: 1680281847651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:27,654 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:27,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49566 deadline: 1680281847652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:27,873 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=581 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/B/81d9f303cbd14bdc9f2575b51e056ea9 2023-03-31 16:56:27,883 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=581 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/C/3a4668a0b2ab46029670fbf3e8649965 2023-03-31 16:56:27,886 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/A/08061d65a0a341e794ff5555c28719cf as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/08061d65a0a341e794ff5555c28719cf 2023-03-31 16:56:27,889 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/08061d65a0a341e794ff5555c28719cf, entries=150, sequenceid=581, filesize=25.4 K 2023-03-31 16:56:27,899 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/B/81d9f303cbd14bdc9f2575b51e056ea9 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/81d9f303cbd14bdc9f2575b51e056ea9 2023-03-31 16:56:27,903 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/81d9f303cbd14bdc9f2575b51e056ea9, entries=150, sequenceid=581, filesize=11.9 K 2023-03-31 16:56:27,904 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/C/3a4668a0b2ab46029670fbf3e8649965 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/3a4668a0b2ab46029670fbf3e8649965 2023-03-31 16:56:27,907 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/3a4668a0b2ab46029670fbf3e8649965, entries=150, sequenceid=581, filesize=11.9 K 2023-03-31 16:56:27,907 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 313b3df11d64a9396698e7102a97394f in 880ms, sequenceid=581, compaction requested=true 2023-03-31 16:56:27,907 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 313b3df11d64a9396698e7102a97394f: 2023-03-31 16:56:27,907 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:27,907 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:27,907 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:27,907 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:27,907 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:27,907 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:27,909 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:27,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:27,909 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:27,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:27,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:27,910 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:27,910 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:27,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:27,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:27,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:27,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:27,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:27,911 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:27,911 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59f76cca[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:27,911 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:27,911 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:27,912 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:27,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:27,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:27,913 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:27,913 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:27,913 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:27,913 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:27,913 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:27,915 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:27,915 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:27,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:27,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:27,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:27,915 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:27,915 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:27,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:27,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:27,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:27,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:27,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:27,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:27,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:27,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:27,918 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:27,918 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:27,918 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:27,918 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:27,918 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:27,918 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:27,919 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:27,920 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:27,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:27,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:27,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:27,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:27,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:27,920 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:27,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:27,920 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:27,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:27,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:27,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:27,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:27,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:27,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:27,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:27,921 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:27,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:27,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:27,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:27,924 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:27,924 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:27,924 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:27,924 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:27,924 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:27,924 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:27,924 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:27,924 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:27,924 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:27,924 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:27,924 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:27,924 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:56:27,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:27,925 DEBUG [Thread-4140] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:27,925 DEBUG [Thread-4140] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:27,924 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:27,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:27,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:27,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:27,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on 313b3df11d64a9396698e7102a97394f 2023-03-31 16:56:27,940 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 313b3df11d64a9396698e7102a97394f 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-03-31 16:56:27,941 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=A 2023-03-31 16:56:27,941 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:56:27,941 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=B 2023-03-31 16:56:27,941 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:56:27,941 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=C 2023-03-31 16:56:27,941 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:56:27,952 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230331cddbbfd90355459e9f5c1ad73be4ec23 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331cddbbfd90355459e9f5c1ad73be4ec23 2023-03-31 16:56:27,957 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:27,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49608 deadline: 1680281847953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:27,957 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:27,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49618 deadline: 1680281847954, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:27,957 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:27,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49602 deadline: 1680281847954, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:27,960 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:27,960 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:27,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49568 deadline: 1680281847956, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:27,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49566 deadline: 1680281847956, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:28,025 DEBUG [Thread-4140] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:28,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:28,027 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:28,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:28,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:28,028 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:28,028 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:28,028 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:28,028 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:28,028 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:28,029 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:28,030 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:28,030 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:28,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:28,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:28,030 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:28,030 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:28,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:28,031 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:28,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:28,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:28,031 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:28,032 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:28,032 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:28,032 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:28,032 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:28,032 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:28,032 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:28,032 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:28,032 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:28,032 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. started... 2023-03-31 16:56:28,032 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. as already flushing 2023-03-31 16:56:28,032 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:28,032 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:28,032 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:28,032 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:28,032 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:28,032 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:28,035 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:28,035 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:28,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:28,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:28,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:28,035 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:28,035 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:28,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:28,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:28,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:28,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:28,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:28,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:28,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@eef79ed[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:28,037 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:28,037 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:28,038 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:28,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:28,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:28,039 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:28,039 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:28,039 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:28,039 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:28,039 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:28,040 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:28,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:28,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:28,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:28,040 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:28,040 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:28,040 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:28,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:28,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:28,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:28,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:28,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:28,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:28,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:28,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:28,042 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:28,042 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:28,042 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:28,043 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:28,043 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:28,043 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:28,044 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:28,044 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:28,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:28,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:28,044 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:28,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:28,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:28,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:28,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:28,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:28,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:28,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:28,045 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:28,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:28,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:28,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:28,046 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:28,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:28,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:28,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:28,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:28,050 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:28,050 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:28,050 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:28,050 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:28,050 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:56:28,050 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:28,050 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:28,050 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:28,050 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:28,050 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:28,050 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:28,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:28,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:28,050 DEBUG [Thread-4140] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:28,050 DEBUG [Thread-4140] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:28,050 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:28,050 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:28,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:28,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:28,061 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:28,061 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:28,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49608 deadline: 1680281848058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:28,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49618 deadline: 1680281848058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:28,150 DEBUG [Thread-4140] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:28,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:28,153 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:28,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:28,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:28,154 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:28,154 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:28,154 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:28,154 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:28,154 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:28,155 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:28,156 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:28,156 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:28,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:28,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:28,156 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:28,156 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:28,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:28,157 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:28,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:28,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:28,157 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:28,157 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:28,158 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:28,158 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:28,158 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:28,158 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:28,158 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:28,158 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:28,158 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:28,158 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. started... 2023-03-31 16:56:28,158 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. as already flushing 2023-03-31 16:56:28,158 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:28,158 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:28,158 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:28,158 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:28,158 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:28,158 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:28,160 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:28,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:28,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:28,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:28,160 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:28,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:28,160 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:28,160 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:28,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:28,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:28,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:28,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:28,162 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:28,162 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20ead450[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:28,162 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:28,162 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:28,164 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:28,164 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:28,164 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:28,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:28,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:28,164 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:28,165 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:28,165 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:28,170 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:28,170 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:28,170 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:28,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:28,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:28,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:28,170 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:28,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:28,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:28,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:28,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:28,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:28,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:28,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:28,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:28,173 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:28,173 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:28,173 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:28,173 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:28,173 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:28,173 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:28,175 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:28,175 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:28,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:28,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:28,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:28,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:28,175 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:28,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:28,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:28,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:28,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:28,176 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:28,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:28,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:28,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:28,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:28,177 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:28,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:28,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:28,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:28,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:28,180 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:28,180 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:28,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:28,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:28,180 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:28,180 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:28,180 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:28,180 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:28,180 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:28,180 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:28,180 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:28,180 DEBUG [Thread-4140] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:28,180 DEBUG [Thread-4140] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:28,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:28,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:28,180 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:28,180 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:56:28,180 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:28,181 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:28,266 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:28,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49608 deadline: 1680281848262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:28,266 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:28,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49618 deadline: 1680281848262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:28,280 DEBUG [Thread-4140] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:28,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:28,282 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:28,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:28,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:28,284 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:28,284 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:28,284 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:28,284 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:28,284 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:28,284 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:28,286 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:28,286 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:28,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:28,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:28,286 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:28,286 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:28,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:28,287 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:28,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:28,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:28,287 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:28,287 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:28,287 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:28,288 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:28,288 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:28,288 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:28,288 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:28,288 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:28,288 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:28,288 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. started... 2023-03-31 16:56:28,288 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. as already flushing 2023-03-31 16:56:28,288 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:28,288 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:28,288 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:28,288 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:28,288 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:28,288 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:28,291 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:28,291 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:28,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:28,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:28,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:28,291 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:28,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:28,291 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:28,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:28,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:28,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:28,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:28,293 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:28,293 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d99a5a1[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:28,293 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:28,293 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:28,294 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:28,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:28,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:28,295 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:28,295 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:28,295 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:28,295 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:28,295 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:28,296 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:28,296 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:28,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:28,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:28,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:28,296 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:28,297 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:28,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:28,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:28,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:28,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:28,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:28,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:28,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:28,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:28,299 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:28,299 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:28,299 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:28,299 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:28,299 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:28,299 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:28,300 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:28,300 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:28,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:28,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:28,300 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:28,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:28,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:28,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:28,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:28,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:28,301 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:28,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:28,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:28,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:28,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:28,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:28,302 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:28,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:28,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:28,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:28,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:28,309 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:28,309 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:28,309 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:28,309 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:28,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:28,309 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:56:28,309 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:28,309 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:28,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:28,309 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:28,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:28,310 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:28,310 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:28,310 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:28,310 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:28,310 DEBUG [Thread-4140] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:28,310 DEBUG [Thread-4140] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:28,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:28,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:28,358 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=619, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/A/59c210a68f5e44ca959bce675f425a8c 2023-03-31 16:56:28,368 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=619 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/B/18ad2b903fb94f4892cd80601d154a44 2023-03-31 16:56:28,379 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=619 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/C/ac262b67896b470491838238721d0b5d 2023-03-31 16:56:28,383 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/A/59c210a68f5e44ca959bce675f425a8c as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/59c210a68f5e44ca959bce675f425a8c 2023-03-31 16:56:28,387 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/59c210a68f5e44ca959bce675f425a8c, entries=150, sequenceid=619, filesize=25.4 K 2023-03-31 16:56:28,388 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/B/18ad2b903fb94f4892cd80601d154a44 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/18ad2b903fb94f4892cd80601d154a44 2023-03-31 16:56:28,391 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/18ad2b903fb94f4892cd80601d154a44, entries=150, sequenceid=619, filesize=11.9 K 2023-03-31 16:56:28,392 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/C/ac262b67896b470491838238721d0b5d as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/ac262b67896b470491838238721d0b5d 2023-03-31 16:56:28,394 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/ac262b67896b470491838238721d0b5d, entries=150, sequenceid=619, filesize=11.9 K 2023-03-31 16:56:28,395 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=40.25 KB/41220 for 313b3df11d64a9396698e7102a97394f in 455ms, sequenceid=619, compaction requested=true 2023-03-31 16:56:28,395 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 313b3df11d64a9396698e7102a97394f: 2023-03-31 16:56:28,395 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:56:28,395 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 16:56:28,395 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 16:56:28,395 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:56:28,395 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:56:28,398 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 105274 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 16:56:28,398 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1912): 313b3df11d64a9396698e7102a97394f/A is initiating minor compaction (all files) 2023-03-31 16:56:28,398 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 313b3df11d64a9396698e7102a97394f/A in TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:28,398 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/700a7c97e96a4bff8157ca4bd033ce14, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/6dc56de547bb4e1b9b26a01c48ab1aae, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/08061d65a0a341e794ff5555c28719cf, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/59c210a68f5e44ca959bce675f425a8c] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp, totalSize=102.8 K 2023-03-31 16:56:28,398 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49834 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 16:56:28,398 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): 313b3df11d64a9396698e7102a97394f/B is initiating minor compaction (all files) 2023-03-31 16:56:28,398 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 313b3df11d64a9396698e7102a97394f/B in TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:28,399 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/f1fd3e829a03446e8ede1c55cbaec48a, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/bfce5e92feec4a8ba92ed249fa70ec5f, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/81d9f303cbd14bdc9f2575b51e056ea9, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/18ad2b903fb94f4892cd80601d154a44] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp, totalSize=48.7 K 2023-03-31 16:56:28,399 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 700a7c97e96a4bff8157ca4bd033ce14, keycount=150, bloomtype=ROW, size=26.5 K, encoding=NONE, compression=NONE, seqNum=532, earliestPutTs=1680281785491 2023-03-31 16:56:28,400 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting f1fd3e829a03446e8ede1c55cbaec48a, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=532, earliestPutTs=1680281785491 2023-03-31 16:56:28,400 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 6dc56de547bb4e1b9b26a01c48ab1aae, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=568, earliestPutTs=1680281785566 2023-03-31 16:56:28,400 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting bfce5e92feec4a8ba92ed249fa70ec5f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=568, earliestPutTs=1680281785566 2023-03-31 16:56:28,400 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 08061d65a0a341e794ff5555c28719cf, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=581, earliestPutTs=1680281786189 2023-03-31 16:56:28,400 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 81d9f303cbd14bdc9f2575b51e056ea9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=581, earliestPutTs=1680281786189 2023-03-31 16:56:28,400 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 59c210a68f5e44ca959bce675f425a8c, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=619, earliestPutTs=1680281787339 2023-03-31 16:56:28,401 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 18ad2b903fb94f4892cd80601d154a44, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=619, earliestPutTs=1680281787339 2023-03-31 16:56:28,410 DEBUG [Thread-4140] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:28,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:28,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:28,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:28,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:28,415 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:28,415 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:28,415 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:28,415 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:28,416 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:28,416 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:28,419 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:28,419 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:28,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:28,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:28,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:28,421 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:28,421 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:28,421 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:28,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:28,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:28,422 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:28,422 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:28,423 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:28,423 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:28,423 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:28,423 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:28,424 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:28,424 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:28,424 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:28,424 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. started... 2023-03-31 16:56:28,424 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 313b3df11d64a9396698e7102a97394f 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-03-31 16:56:28,425 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=A 2023-03-31 16:56:28,425 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:56:28,425 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=B 2023-03-31 16:56:28,425 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:56:28,425 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=C 2023-03-31 16:56:28,425 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:56:28,429 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): 313b3df11d64a9396698e7102a97394f#B#compaction#669 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:56:28,437 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] throttle.PressureAwareThroughputController(145): 313b3df11d64a9396698e7102a97394f#A#compaction#668 average throughput is 1.97 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:56:28,461 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023033117507ce9012548ad877156ac46d97008 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033117507ce9012548ad877156ac46d97008 2023-03-31 16:56:28,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on 313b3df11d64a9396698e7102a97394f 2023-03-31 16:56:28,463 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. as already flushing 2023-03-31 16:56:28,487 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680281788472, max timestamp=1680281788485Num uniques 859; 2023-03-31 16:56:28,487 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 16:56:28,487 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680281788472, max timestamp=1680281788485Num uniques 859; 2023-03-31 16:56:28,487 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-31 16:56:28,489 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680281788479, max timestamp=1680281788485Num uniques 859; 2023-03-31 16:56:28,489 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 16:56:28,516 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:28,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49566 deadline: 1680281848512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:28,516 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:28,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49602 deadline: 1680281848513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:28,520 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:28,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49568 deadline: 1680281848515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:28,568 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:28,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49618 deadline: 1680281848567, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:28,574 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:28,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49608 deadline: 1680281848569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:28,621 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:28,621 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:28,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49602 deadline: 1680281848617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:28,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49566 deadline: 1680281848617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:28,623 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:28,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49568 deadline: 1680281848621, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:28,823 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:28,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49602 deadline: 1680281848822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:28,824 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:28,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49566 deadline: 1680281848823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:28,828 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:28,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49568 deadline: 1680281848825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:28,846 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/B/fe49e2c7eadd43cda86e40eacec6c4bf as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/fe49e2c7eadd43cda86e40eacec6c4bf 2023-03-31 16:56:28,849 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 313b3df11d64a9396698e7102a97394f/B of 313b3df11d64a9396698e7102a97394f into fe49e2c7eadd43cda86e40eacec6c4bf(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:56:28,850 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 313b3df11d64a9396698e7102a97394f: 2023-03-31 16:56:28,850 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f., storeName=313b3df11d64a9396698e7102a97394f/B, priority=12, startTime=1680281788395; duration=0sec 2023-03-31 16:56:28,850 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:56:28,850 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 16:56:28,851 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49834 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 16:56:28,852 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): 313b3df11d64a9396698e7102a97394f/C is initiating minor compaction (all files) 2023-03-31 16:56:28,852 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 313b3df11d64a9396698e7102a97394f/C in TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:28,852 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/f691f015cb3a43c3bf17fd7ca7b13f0f, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/6dd9689ef20e41e08bc9f56a4c194358, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/3a4668a0b2ab46029670fbf3e8649965, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/ac262b67896b470491838238721d0b5d] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp, totalSize=48.7 K 2023-03-31 16:56:28,852 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting f691f015cb3a43c3bf17fd7ca7b13f0f, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=532, earliestPutTs=1680281785491 2023-03-31 16:56:28,853 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 6dd9689ef20e41e08bc9f56a4c194358, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=568, earliestPutTs=1680281785566 2023-03-31 16:56:28,853 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 3a4668a0b2ab46029670fbf3e8649965, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=581, earliestPutTs=1680281786189 2023-03-31 16:56:28,853 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting ac262b67896b470491838238721d0b5d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=619, earliestPutTs=1680281787339 2023-03-31 16:56:28,861 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): 313b3df11d64a9396698e7102a97394f#C#compaction#671 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:56:28,868 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/C/e5dbf7fcfcba48d2af9a88d96a923aac as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/e5dbf7fcfcba48d2af9a88d96a923aac 2023-03-31 16:56:28,872 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 313b3df11d64a9396698e7102a97394f/C of 313b3df11d64a9396698e7102a97394f into e5dbf7fcfcba48d2af9a88d96a923aac(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:56:28,872 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 313b3df11d64a9396698e7102a97394f: 2023-03-31 16:56:28,872 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f., storeName=313b3df11d64a9396698e7102a97394f/C, priority=12, startTime=1680281788395; duration=0sec 2023-03-31 16:56:28,872 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:56:28,882 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=628, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/A/430bbd9a2cec43db942bfa4796567777 2023-03-31 16:56:29,071 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:29,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49618 deadline: 1680281849070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:29,078 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:29,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49608 deadline: 1680281849077, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:29,127 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:29,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49602 deadline: 1680281849127, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:29,127 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:29,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49566 deadline: 1680281849127, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:29,132 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:29,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49568 deadline: 1680281849130, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:29,263 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/A/1022e0ff14a94922adf2d183f0b3d968 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/1022e0ff14a94922adf2d183f0b3d968 2023-03-31 16:56:29,267 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 313b3df11d64a9396698e7102a97394f/A of 313b3df11d64a9396698e7102a97394f into 1022e0ff14a94922adf2d183f0b3d968(size=26.6 K), total size for store is 26.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:56:29,267 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 313b3df11d64a9396698e7102a97394f: 2023-03-31 16:56:29,267 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f., storeName=313b3df11d64a9396698e7102a97394f/A, priority=12, startTime=1680281788395; duration=0sec 2023-03-31 16:56:29,268 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:56:29,289 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=628 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/B/68dfb52b2a264229be666c4a8e841a64 2023-03-31 16:56:29,630 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:29,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49602 deadline: 1680281849628, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:29,634 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:29,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49566 deadline: 1680281849631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:29,637 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:29,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49568 deadline: 1680281849635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:29,698 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=628 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/C/4c1f9ca92df04bf3b1a7c3153dee20aa 2023-03-31 16:56:29,702 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/A/430bbd9a2cec43db942bfa4796567777 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/430bbd9a2cec43db942bfa4796567777 2023-03-31 16:56:29,706 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/430bbd9a2cec43db942bfa4796567777, entries=150, sequenceid=628, filesize=25.4 K 2023-03-31 16:56:29,706 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/B/68dfb52b2a264229be666c4a8e841a64 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/68dfb52b2a264229be666c4a8e841a64 2023-03-31 16:56:29,709 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/68dfb52b2a264229be666c4a8e841a64, entries=150, sequenceid=628, filesize=11.9 K 2023-03-31 16:56:29,710 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/C/4c1f9ca92df04bf3b1a7c3153dee20aa as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/4c1f9ca92df04bf3b1a7c3153dee20aa 2023-03-31 16:56:29,722 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/4c1f9ca92df04bf3b1a7c3153dee20aa, entries=150, sequenceid=628, filesize=11.9 K 2023-03-31 16:56:29,722 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=138.65 KB/141980 for 313b3df11d64a9396698e7102a97394f in 1298ms, sequenceid=628, compaction requested=false 2023-03-31 16:56:29,722 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 313b3df11d64a9396698e7102a97394f: 2023-03-31 16:56:29,722 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:29,722 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:29,723 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:29,723 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:29,723 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:29,723 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:29,724 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:29,724 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:29,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:29,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:29,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:29,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:29,725 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:29,725 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:29,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:29,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:29,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:29,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:29,727 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:29,727 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f6a09e4[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:29,727 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:29,727 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:29,729 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:29,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:29,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:29,729 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:29,729 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:29,729 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:29,729 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:29,729 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:29,731 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:29,731 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:29,731 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:29,731 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:29,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:29,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:29,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:29,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:29,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:29,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:29,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:29,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:29,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:29,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:29,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:29,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:29,734 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:29,734 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:29,734 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:29,734 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:29,734 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:29,736 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:29,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:29,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:29,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:29,736 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:29,736 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:29,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:29,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:29,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:29,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:29,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:29,737 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:29,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:29,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:29,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:29,739 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:29,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:29,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:29,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:29,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:29,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:29,741 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:29,741 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:29,741 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:29,741 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:29,741 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:29,741 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:29,741 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:29,742 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:29,741 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:29,742 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:29,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:29,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:29,741 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:29,742 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:29,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:29,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:29,742 DEBUG [Thread-4140] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:29,742 DEBUG [Thread-4140] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:29,742 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:56:29,842 DEBUG [Thread-4140] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:29,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:29,844 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:29,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:29,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:29,846 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:29,846 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:29,846 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:29,846 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:29,847 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:29,847 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:29,848 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:29,848 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:29,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:29,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:29,849 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:29,849 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:29,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:29,849 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:29,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:29,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:29,849 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:29,850 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:29,850 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:29,850 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:29,850 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:29,850 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:29,850 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:29,850 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:29,850 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:29,850 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. started... 2023-03-31 16:56:29,850 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 313b3df11d64a9396698e7102a97394f 3/3 column families, dataSize=138.65 KB heapSize=395.14 KB 2023-03-31 16:56:29,851 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=A 2023-03-31 16:56:29,851 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:56:29,851 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=B 2023-03-31 16:56:29,851 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:56:29,851 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=C 2023-03-31 16:56:29,851 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:56:29,871 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303319e8a55c96dd04286ac791daad77ad965 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303319e8a55c96dd04286ac791daad77ad965 2023-03-31 16:56:29,882 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=668, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/A/931939c421d443f3b847e296227ce9e5 2023-03-31 16:56:30,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on 313b3df11d64a9396698e7102a97394f 2023-03-31 16:56:30,083 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. as already flushing 2023-03-31 16:56:30,095 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:30,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49608 deadline: 1680281850092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,095 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:30,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49618 deadline: 1680281850094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,198 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:30,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49618 deadline: 1680281850196, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,198 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:30,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49608 deadline: 1680281850196, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,294 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=668 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/B/36340866c5b7411389d1e2b6045cecc5 2023-03-31 16:56:30,312 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=668 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/C/cd1bbcd079304c228ff1a883b3d220b6 2023-03-31 16:56:30,316 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/A/931939c421d443f3b847e296227ce9e5 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/931939c421d443f3b847e296227ce9e5 2023-03-31 16:56:30,319 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/931939c421d443f3b847e296227ce9e5, entries=150, sequenceid=668, filesize=25.4 K 2023-03-31 16:56:30,322 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/B/36340866c5b7411389d1e2b6045cecc5 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/36340866c5b7411389d1e2b6045cecc5 2023-03-31 16:56:30,325 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/36340866c5b7411389d1e2b6045cecc5, entries=150, sequenceid=668, filesize=11.9 K 2023-03-31 16:56:30,326 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/C/cd1bbcd079304c228ff1a883b3d220b6 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/cd1bbcd079304c228ff1a883b3d220b6 2023-03-31 16:56:30,329 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/cd1bbcd079304c228ff1a883b3d220b6, entries=150, sequenceid=668, filesize=11.9 K 2023-03-31 16:56:30,329 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~138.65 KB/141980, heapSize ~395.09 KB/404576, currentSize=40.25 KB/41220 for 313b3df11d64a9396698e7102a97394f in 479ms, sequenceid=668, compaction requested=true 2023-03-31 16:56:30,329 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 313b3df11d64a9396698e7102a97394f: 2023-03-31 16:56:30,329 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:30,330 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:30,330 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:30,330 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,330 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:30,330 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:30,332 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:30,332 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:30,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:30,333 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:30,333 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:30,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:30,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:30,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:30,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:30,334 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:30,334 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a3f35f5[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:30,334 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:30,334 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:30,336 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:30,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:30,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:30,336 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:30,336 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:30,336 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:30,336 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,336 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:30,337 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,337 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:30,337 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:30,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:30,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:30,337 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:30,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:30,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:30,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:30,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:30,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:30,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:30,340 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:30,340 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:30,340 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:30,340 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:30,340 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:30,340 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:30,345 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:30,345 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:30,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:30,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:30,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:30,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:30,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:30,346 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:30,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:30,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:30,346 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:30,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:30,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:30,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:30,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:30,348 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:30,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:30,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,351 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:30,351 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:30,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:30,351 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:30,351 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:30,351 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:30,351 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:30,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:30,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:30,351 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:30,351 DEBUG [Thread-4140] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:30,351 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:30,351 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,352 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:30,352 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:30,352 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:30,351 DEBUG [Thread-4140] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:30,352 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:56:30,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on 313b3df11d64a9396698e7102a97394f 2023-03-31 16:56:30,404 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 313b3df11d64a9396698e7102a97394f 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-31 16:56:30,404 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=A 2023-03-31 16:56:30,405 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:56:30,405 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=B 2023-03-31 16:56:30,405 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:56:30,405 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=C 2023-03-31 16:56:30,405 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:56:30,420 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230331c5242b245190417baf1d747a2d95b78c to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331c5242b245190417baf1d747a2d95b78c 2023-03-31 16:56:30,425 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=678, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/A/d3b73918b77448d1bbc2617c1921c17d 2023-03-31 16:56:30,434 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=678 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/B/28151afcbff7410396fe650903aae418 2023-03-31 16:56:30,445 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=678 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/C/b263025817544911894ba37ce2d9f0c5 2023-03-31 16:56:30,449 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/A/d3b73918b77448d1bbc2617c1921c17d as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/d3b73918b77448d1bbc2617c1921c17d 2023-03-31 16:56:30,452 DEBUG [Thread-4140] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:30,452 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/d3b73918b77448d1bbc2617c1921c17d, entries=150, sequenceid=678, filesize=25.4 K 2023-03-31 16:56:30,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:30,452 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/B/28151afcbff7410396fe650903aae418 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/28151afcbff7410396fe650903aae418 2023-03-31 16:56:30,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:30,456 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680281790441, max timestamp=1680281790453Num uniques 859; 2023-03-31 16:56:30,456 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 16:56:30,456 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680281790441, max timestamp=1680281790453Num uniques 859; 2023-03-31 16:56:30,456 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680281790441, max timestamp=1680281790453Num uniques 859; 2023-03-31 16:56:30,456 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 16:56:30,456 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 16:56:30,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:30,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:30,457 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:30,457 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:30,457 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:30,457 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:30,457 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:30,457 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/28151afcbff7410396fe650903aae418, entries=100, sequenceid=678, filesize=9.5 K 2023-03-31 16:56:30,457 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:30,458 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/C/b263025817544911894ba37ce2d9f0c5 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/b263025817544911894ba37ce2d9f0c5 2023-03-31 16:56:30,459 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:30,459 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:30,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:30,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:30,459 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,459 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:30,460 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:30,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:30,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:30,460 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:30,460 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:30,461 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:30,461 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:30,461 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:30,461 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:30,461 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/b263025817544911894ba37ce2d9f0c5, entries=100, sequenceid=678, filesize=9.5 K 2023-03-31 16:56:30,462 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:30,462 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=46.96 KB/48090 for 313b3df11d64a9396698e7102a97394f in 58ms, sequenceid=678, compaction requested=true 2023-03-31 16:56:30,462 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:30,462 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 313b3df11d64a9396698e7102a97394f: 2023-03-31 16:56:30,462 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. started... 2023-03-31 16:56:30,462 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:30,462 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:56:30,462 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 313b3df11d64a9396698e7102a97394f 3/3 column families, dataSize=46.96 KB heapSize=126 KB 2023-03-31 16:56:30,462 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:56:30,462 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 16:56:30,462 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=A 2023-03-31 16:56:30,462 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:56:30,462 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:56:30,462 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-31 16:56:30,463 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=B 2023-03-31 16:56:30,463 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:56:30,463 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=C 2023-03-31 16:56:30,463 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:56:30,464 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 105406 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 16:56:30,464 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47526 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-31 16:56:30,464 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1912): 313b3df11d64a9396698e7102a97394f/A is initiating minor compaction (all files) 2023-03-31 16:56:30,464 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): 313b3df11d64a9396698e7102a97394f/B is initiating minor compaction (all files) 2023-03-31 16:56:30,464 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 313b3df11d64a9396698e7102a97394f/A in TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:30,464 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 313b3df11d64a9396698e7102a97394f/B in TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:30,464 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/1022e0ff14a94922adf2d183f0b3d968, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/430bbd9a2cec43db942bfa4796567777, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/931939c421d443f3b847e296227ce9e5, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/d3b73918b77448d1bbc2617c1921c17d] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp, totalSize=102.9 K 2023-03-31 16:56:30,464 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/fe49e2c7eadd43cda86e40eacec6c4bf, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/68dfb52b2a264229be666c4a8e841a64, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/36340866c5b7411389d1e2b6045cecc5, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/28151afcbff7410396fe650903aae418] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp, totalSize=46.4 K 2023-03-31 16:56:30,465 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 1022e0ff14a94922adf2d183f0b3d968, keycount=150, bloomtype=ROW, size=26.6 K, encoding=NONE, compression=NONE, seqNum=619, earliestPutTs=1680281787339 2023-03-31 16:56:30,465 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting fe49e2c7eadd43cda86e40eacec6c4bf, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=619, earliestPutTs=1680281787339 2023-03-31 16:56:30,465 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 430bbd9a2cec43db942bfa4796567777, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=628, earliestPutTs=1680281787948 2023-03-31 16:56:30,465 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 68dfb52b2a264229be666c4a8e841a64, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=628, earliestPutTs=1680281787948 2023-03-31 16:56:30,465 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 931939c421d443f3b847e296227ce9e5, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=668, earliestPutTs=1680281788497 2023-03-31 16:56:30,465 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 36340866c5b7411389d1e2b6045cecc5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=668, earliestPutTs=1680281788497 2023-03-31 16:56:30,466 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 28151afcbff7410396fe650903aae418, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=678, earliestPutTs=1680281790091 2023-03-31 16:56:30,466 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting d3b73918b77448d1bbc2617c1921c17d, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=678, earliestPutTs=1680281790091 2023-03-31 16:56:30,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on 313b3df11d64a9396698e7102a97394f 2023-03-31 16:56:30,467 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. as already flushing 2023-03-31 16:56:30,474 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): 313b3df11d64a9396698e7102a97394f#B#compaction#681 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:56:30,486 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] throttle.PressureAwareThroughputController(145): 313b3df11d64a9396698e7102a97394f#A#compaction#682 average throughput is 3.29 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:56:30,493 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303319ca1fa3ff55f462e94dcf6230fc23a15 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303319ca1fa3ff55f462e94dcf6230fc23a15 2023-03-31 16:56:30,501 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=702, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/A/4804b31b8abf48a09e72df1150feab1c 2023-03-31 16:56:30,511 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680281790487, max timestamp=1680281790509Num uniques 859; 2023-03-31 16:56:30,512 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 16:56:30,512 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680281790487, max timestamp=1680281790509Num uniques 859; 2023-03-31 16:56:30,512 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680281790487, max timestamp=1680281790509Num uniques 859; 2023-03-31 16:56:30,512 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 16:56:30,512 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 16:56:30,544 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=702 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/B/ef2d22bd65f84af09b5a784b002cc2fd 2023-03-31 16:56:30,553 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=702 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/C/8b7a3c94058a4ba09c03af8d5ae409f3 2023-03-31 16:56:30,557 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/A/4804b31b8abf48a09e72df1150feab1c as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/4804b31b8abf48a09e72df1150feab1c 2023-03-31 16:56:30,559 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680281790542, max timestamp=1680281790556Num uniques 1018; 2023-03-31 16:56:30,559 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680281790542, max timestamp=1680281790556Num uniques 1018; 2023-03-31 16:56:30,559 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-03-31 16:56:30,559 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-03-31 16:56:30,559 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680281790542, max timestamp=1680281790556Num uniques 1018; 2023-03-31 16:56:30,559 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-03-31 16:56:30,560 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/4804b31b8abf48a09e72df1150feab1c, entries=150, sequenceid=702, filesize=25.4 K 2023-03-31 16:56:30,561 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/B/ef2d22bd65f84af09b5a784b002cc2fd as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/ef2d22bd65f84af09b5a784b002cc2fd 2023-03-31 16:56:30,564 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/ef2d22bd65f84af09b5a784b002cc2fd, entries=150, sequenceid=702, filesize=11.9 K 2023-03-31 16:56:30,565 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/C/8b7a3c94058a4ba09c03af8d5ae409f3 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/8b7a3c94058a4ba09c03af8d5ae409f3 2023-03-31 16:56:30,567 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/8b7a3c94058a4ba09c03af8d5ae409f3, entries=150, sequenceid=702, filesize=11.9 K 2023-03-31 16:56:30,568 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~125.95 KB/128976, currentSize=53.67 KB/54960 for 313b3df11d64a9396698e7102a97394f in 106ms, sequenceid=702, compaction requested=true 2023-03-31 16:56:30,568 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 313b3df11d64a9396698e7102a97394f: 2023-03-31 16:56:30,568 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:30,568 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:30,568 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:30,568 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,568 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:30,568 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:30,570 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,570 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:30,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:30,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:30,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on 313b3df11d64a9396698e7102a97394f 2023-03-31 16:56:30,570 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 313b3df11d64a9396698e7102a97394f 3/3 column families, dataSize=60.38 KB heapSize=160.31 KB 2023-03-31 16:56:30,570 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:30,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:30,570 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:30,570 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=A 2023-03-31 16:56:30,571 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:56:30,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:30,571 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=B 2023-03-31 16:56:30,571 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:56:30,571 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=C 2023-03-31 16:56:30,571 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:56:30,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:30,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:30,573 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:30,573 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c012ab6[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:30,573 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:30,573 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:30,576 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,577 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:30,578 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:30,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:30,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:30,578 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:30,578 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:30,578 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:30,580 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,580 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:30,580 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:30,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:30,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:30,580 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:30,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:30,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:30,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:30,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:30,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:30,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:30,583 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:30,583 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:30,583 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:30,583 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:30,583 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:30,583 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:30,585 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:30,585 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:30,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:30,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:30,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:30,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:30,585 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:30,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:30,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:30,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:30,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:30,585 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:30,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:30,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:30,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:30,587 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:30,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:30,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,589 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,589 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:30,589 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:30,589 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:30,589 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:30,589 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:30,589 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:56:30,589 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:30,589 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:30,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:30,590 DEBUG [Thread-4140] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:30,590 DEBUG [Thread-4140] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:30,589 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:30,589 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:30,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:30,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:30,590 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:30,590 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:30,597 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303317702cc5413a0447d95c9eac72b2d0cb5 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303317702cc5413a0447d95c9eac72b2d0cb5 2023-03-31 16:56:30,615 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:30,615 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680281790594, max timestamp=1680281790613Num uniques 859; 2023-03-31 16:56:30,615 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680281790594, max timestamp=1680281790613Num uniques 859; 2023-03-31 16:56:30,615 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 16:56:30,615 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680281790594, max timestamp=1680281790613Num uniques 859; 2023-03-31 16:56:30,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49618 deadline: 1680281850611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,615 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 16:56:30,615 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-31 16:56:30,654 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:30,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49568 deadline: 1680281850648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,657 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:30,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49566 deadline: 1680281850652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,657 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:30,657 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:30,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49602 deadline: 1680281850654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49608 deadline: 1680281850653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,690 DEBUG [Thread-4140] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:30,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:30,692 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:30,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:30,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:30,693 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:30,693 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:30,693 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:30,693 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:30,694 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:30,694 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:30,696 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:30,696 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:30,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:30,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:30,697 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,697 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:30,697 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:30,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:30,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:30,698 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:30,698 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:30,698 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:30,698 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:30,698 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:30,698 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:30,698 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:30,698 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:30,698 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:30,698 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. started... 2023-03-31 16:56:30,698 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. as already flushing 2023-03-31 16:56:30,699 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:30,699 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:30,699 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:30,699 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,699 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:30,699 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:30,700 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,700 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:30,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:30,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:30,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:30,701 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:30,701 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:30,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:30,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:30,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:30,703 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:30,703 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c753e3a[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:30,703 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:30,703 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:30,704 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:30,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:30,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:30,704 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:30,704 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:30,704 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:30,704 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,704 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:30,705 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:30,705 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:30,706 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:30,706 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:30,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:30,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:30,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:30,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:30,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:30,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:30,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:30,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:30,708 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:30,708 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:30,708 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:30,708 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:30,708 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:30,711 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:30,711 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:30,711 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:30,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:30,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:30,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:30,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:30,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:30,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:30,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:30,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:30,711 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:30,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:30,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:30,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:30,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,713 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:30,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:30,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,715 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:30,715 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,715 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:30,715 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:30,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:30,715 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:56:30,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:30,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:30,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:30,715 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:30,715 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:30,716 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,716 DEBUG [Thread-4140] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:30,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:30,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:30,716 DEBUG [Thread-4140] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:30,716 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:30,716 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:30,716 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:30,719 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:30,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49618 deadline: 1680281850717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,757 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:30,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49568 deadline: 1680281850755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,763 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:30,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49566 deadline: 1680281850758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,763 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:30,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49608 deadline: 1680281850758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,763 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:30,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49602 deadline: 1680281850758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,816 DEBUG [Thread-4140] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:30,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:30,818 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:30,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:30,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:30,819 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:30,819 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:30,819 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:30,819 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:30,820 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:30,820 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:30,821 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,821 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:30,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:30,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:30,822 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,822 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:30,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:30,822 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:30,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:30,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:30,823 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:30,823 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:30,823 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:30,823 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:30,823 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:30,823 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:30,823 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:30,823 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:30,823 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:30,823 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. started... 2023-03-31 16:56:30,824 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. as already flushing 2023-03-31 16:56:30,824 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:30,824 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:30,824 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:30,824 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,824 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:30,824 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:30,826 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,826 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:30,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:30,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:30,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:30,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:30,826 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:30,826 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:30,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:30,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:30,827 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:30,827 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@136f2779[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:30,827 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:30,827 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:30,830 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:30,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:30,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:30,830 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:30,830 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:30,830 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:30,830 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,830 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:30,832 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,832 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:30,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:30,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:30,832 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:30,832 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:30,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:30,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:30,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:30,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:30,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:30,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:30,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:30,834 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:30,834 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:30,834 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:30,834 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:30,834 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:30,836 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:30,836 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:30,836 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:30,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:30,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:30,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:30,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:30,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:30,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:30,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:30,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:30,837 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:30,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:30,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:30,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:30,838 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:30,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:30,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,844 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,844 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:30,844 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:30,844 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,844 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:30,844 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:30,844 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:30,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:30,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:30,844 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:30,844 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:30,844 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:30,844 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:56:30,844 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:30,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:30,845 DEBUG [Thread-4140] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:30,845 DEBUG [Thread-4140] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:30,845 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:30,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:30,893 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/B/81ce038b5a8e4456a5638120e711c97c as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/81ce038b5a8e4456a5638120e711c97c 2023-03-31 16:56:30,898 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 313b3df11d64a9396698e7102a97394f/B of 313b3df11d64a9396698e7102a97394f into 81ce038b5a8e4456a5638120e711c97c(size=13.2 K), total size for store is 25.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:56:30,898 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 313b3df11d64a9396698e7102a97394f: 2023-03-31 16:56:30,898 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f., storeName=313b3df11d64a9396698e7102a97394f/B, priority=12, startTime=1680281790462; duration=0sec 2023-03-31 16:56:30,898 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:56:30,898 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-31 16:56:30,900 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59712 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-31 16:56:30,900 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): 313b3df11d64a9396698e7102a97394f/C is initiating minor compaction (all files) 2023-03-31 16:56:30,900 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 313b3df11d64a9396698e7102a97394f/C in TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:30,900 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/e5dbf7fcfcba48d2af9a88d96a923aac, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/4c1f9ca92df04bf3b1a7c3153dee20aa, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/cd1bbcd079304c228ff1a883b3d220b6, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/b263025817544911894ba37ce2d9f0c5, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/8b7a3c94058a4ba09c03af8d5ae409f3] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp, totalSize=58.3 K 2023-03-31 16:56:30,900 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting e5dbf7fcfcba48d2af9a88d96a923aac, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=619, earliestPutTs=1680281787339 2023-03-31 16:56:30,901 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 4c1f9ca92df04bf3b1a7c3153dee20aa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=628, earliestPutTs=1680281787948 2023-03-31 16:56:30,901 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting cd1bbcd079304c228ff1a883b3d220b6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=668, earliestPutTs=1680281788497 2023-03-31 16:56:30,901 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting b263025817544911894ba37ce2d9f0c5, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=678, earliestPutTs=1680281790091 2023-03-31 16:56:30,902 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 8b7a3c94058a4ba09c03af8d5ae409f3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=702, earliestPutTs=1680281790447 2023-03-31 16:56:30,908 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): 313b3df11d64a9396698e7102a97394f#C#compaction#686 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:56:30,915 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/C/00ce0de93a254e8fac8937297e525720 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/00ce0de93a254e8fac8937297e525720 2023-03-31 16:56:30,918 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 313b3df11d64a9396698e7102a97394f/C of 313b3df11d64a9396698e7102a97394f into 00ce0de93a254e8fac8937297e525720(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:56:30,918 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 313b3df11d64a9396698e7102a97394f: 2023-03-31 16:56:30,918 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f., storeName=313b3df11d64a9396698e7102a97394f/C, priority=11, startTime=1680281790462; duration=0sec 2023-03-31 16:56:30,918 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:56:30,924 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:30,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49618 deadline: 1680281850921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,945 DEBUG [Thread-4140] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:30,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:30,947 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:30,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:30,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:30,948 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:30,948 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:30,948 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:30,948 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:30,948 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:30,948 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:30,950 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,950 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:30,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:30,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:30,950 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,950 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:30,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:30,951 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:30,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:30,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:30,951 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:30,951 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:30,951 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:30,952 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:30,952 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:30,952 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:30,952 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:30,952 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:30,952 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:30,952 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. started... 2023-03-31 16:56:30,952 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. as already flushing 2023-03-31 16:56:30,952 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:30,952 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:30,952 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:30,952 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,952 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:30,952 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:30,954 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,954 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:30,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:30,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:30,955 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:30,955 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:30,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:30,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:30,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:30,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:30,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:30,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@701c387c[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:30,956 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:30,956 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:30,957 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:30,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:30,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:30,958 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:30,958 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:30,958 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:30,958 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,958 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:30,960 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,960 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:30,960 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:30,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:30,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:30,960 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:30,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:30,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:30,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:30,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:30,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:30,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:30,963 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:30,963 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:30,963 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:30,963 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:30,963 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:30,963 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:30,964 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:30,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49568 deadline: 1680281850959, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,965 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:30,965 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:30,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:30,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:30,965 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:30,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:30,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:30,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:30,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:30,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:30,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:30,966 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:30,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:30,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:30,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:30,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,967 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:30,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:30,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,968 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:30,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49608 deadline: 1680281850964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,968 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:30,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49566 deadline: 1680281850965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,969 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:30,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49602 deadline: 1680281850965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,969 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,969 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:30,969 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:30,969 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:30,969 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:30,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:30,969 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:30,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:30,969 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:30,969 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:30,969 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:30,969 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:30,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:30,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:30,970 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:30,970 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:30,970 DEBUG [Thread-4140] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:30,970 DEBUG [Thread-4140] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:31,002 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=744, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/A/53ee387229a547ecbaf01b7bce339d15 2023-03-31 16:56:31,070 DEBUG [Thread-4140] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:31,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:31,072 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:31,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:31,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:31,073 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:31,073 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:31,074 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:31,074 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:31,074 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:31,074 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:31,075 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,076 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:31,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:31,076 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,076 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:31,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:31,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:31,076 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:31,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:31,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:31,077 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:31,077 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:31,077 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:31,077 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:31,077 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:31,077 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:31,077 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:31,077 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:31,077 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:31,077 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. started... 2023-03-31 16:56:31,077 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. as already flushing 2023-03-31 16:56:31,077 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:31,077 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:31,077 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:31,077 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,077 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:31,078 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:31,080 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:31,080 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:31,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:31,080 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:31,080 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:31,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:31,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:31,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:31,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:31,082 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:31,082 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@152df853[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:31,082 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:31,082 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:31,083 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:31,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:31,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:31,084 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,084 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:31,084 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:31,084 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:31,084 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:31,085 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:31,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:31,086 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:31,086 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:31,086 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:31,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:31,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:31,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:31,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:31,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:31,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:31,088 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:31,088 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:31,088 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:31,088 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:31,088 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:31,088 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:31,090 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:31,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:31,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:31,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:31,090 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:31,091 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:31,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:31,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:31,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:31,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:31,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:31,091 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:31,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:31,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:31,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:31,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,093 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:31,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:31,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,097 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,097 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:31,097 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:31,097 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:31,097 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:31,097 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:31,097 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:31,097 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,097 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:31,097 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:31,097 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:31,097 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:31,097 DEBUG [Thread-4140] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:31,097 DEBUG [Thread-4140] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:31,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:31,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:31,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:31,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:31,197 DEBUG [Thread-4140] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:31,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:31,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:31,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:31,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:31,201 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:31,201 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:31,201 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:31,201 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:31,201 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:31,201 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:31,202 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,202 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:31,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:31,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:31,203 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,203 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:31,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:31,204 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:31,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:31,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:31,204 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:31,204 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:31,204 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:31,204 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:31,204 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:31,205 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:31,205 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:31,205 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:31,205 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:31,205 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. started... 2023-03-31 16:56:31,205 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. as already flushing 2023-03-31 16:56:31,205 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:31,205 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:31,205 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:31,205 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,205 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:31,205 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:31,207 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,207 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:31,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:31,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:31,207 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:31,207 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:31,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:31,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:31,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:31,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:31,209 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:31,209 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10ac9022[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:31,209 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:31,209 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:31,210 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:31,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:31,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:31,210 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:31,210 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:31,210 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:31,210 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,210 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:31,213 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:31,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:31,213 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:31,213 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:31,213 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:31,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:31,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:31,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:31,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:31,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:31,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:31,215 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:31,215 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:31,215 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:31,215 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:31,215 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:31,215 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:31,217 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:31,217 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:31,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:31,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:31,217 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:31,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:31,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:31,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:31,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:31,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:31,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:31,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:31,218 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:31,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:31,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:31,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,220 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:31,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:31,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,222 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:31,222 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,222 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:31,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:31,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:31,222 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:31,222 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:31,223 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:31,223 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,223 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:31,223 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:31,223 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:31,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:31,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:31,223 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:31,223 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:31,223 DEBUG [Thread-4140] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:31,223 DEBUG [Thread-4140] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:31,227 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:31,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49618 deadline: 1680281851225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,268 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:31,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49568 deadline: 1680281851266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,272 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:31,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49608 deadline: 1680281851270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,272 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:31,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49566 deadline: 1680281851270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,273 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:31,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33441] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49602 deadline: 1680281851271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,304 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/A/66b68c69223946c49982503f26de7a5e as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/66b68c69223946c49982503f26de7a5e 2023-03-31 16:56:31,310 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 313b3df11d64a9396698e7102a97394f/A of 313b3df11d64a9396698e7102a97394f into 66b68c69223946c49982503f26de7a5e(size=26.8 K), total size for store is 52.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:56:31,310 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 313b3df11d64a9396698e7102a97394f: 2023-03-31 16:56:31,310 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f., storeName=313b3df11d64a9396698e7102a97394f/A, priority=12, startTime=1680281790462; duration=0sec 2023-03-31 16:56:31,310 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:56:31,323 DEBUG [Thread-4140] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:31,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:31,325 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:31,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:31,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:31,327 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:31,327 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:31,327 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:31,327 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:31,327 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:31,327 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:31,333 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:31,333 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:31,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:31,333 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,333 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:31,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:31,334 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:31,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:31,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:31,334 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:31,334 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:31,334 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:31,334 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:31,335 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:31,335 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:31,335 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:31,335 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:31,335 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:31,335 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. started... 2023-03-31 16:56:31,335 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. as already flushing 2023-03-31 16:56:31,335 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:31,335 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:31,335 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:31,335 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,335 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:31,335 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:31,337 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,337 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:31,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:31,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:31,337 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:31,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:31,337 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:31,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:31,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:31,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:31,339 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:31,339 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a1229a1[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:31,339 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:31,339 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:31,340 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:31,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:31,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:31,340 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,340 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:31,340 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:31,340 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:31,340 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:31,341 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:31,341 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:31,341 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,341 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:31,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:31,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:31,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:31,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:31,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:31,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:31,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:31,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:31,344 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:31,344 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:31,344 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:31,344 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:31,344 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:31,345 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:31,347 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:31,347 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:31,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:31,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:31,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:31,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:31,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:31,347 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:31,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:31,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:31,348 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:31,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:31,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:31,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:31,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:31,349 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:31,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:31,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,351 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,351 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:31,351 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:31,351 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,351 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:56:31,351 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:31,351 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:31,351 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:31,351 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:31,351 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:31,351 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:31,352 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:31,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:31,352 DEBUG [Thread-4140] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:31,351 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:31,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:31,352 DEBUG [Thread-4140] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:31,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:31,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:31,411 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=744 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/B/1df5ff3f21244ada9d27c98c3d14937d 2023-03-31 16:56:31,452 DEBUG [Thread-4140] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:31,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:31,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:31,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:31,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:31,455 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:31,455 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:31,455 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:31,455 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:31,456 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:31,456 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:31,457 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:31,457 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:31,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:31,457 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,457 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:31,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:31,458 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:31,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:31,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:31,458 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:31,458 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:31,459 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:31,459 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:31,459 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:31,459 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:31,459 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:31,459 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:31,459 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:31,459 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. started... 2023-03-31 16:56:31,459 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. as already flushing 2023-03-31 16:56:31,459 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:31,459 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:31,459 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:31,459 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,459 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:31,459 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:31,461 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,461 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:31,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:31,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:31,462 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:31,462 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:31,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:31,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:31,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:31,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:31,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:31,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b5c3147[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:31,463 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:31,463 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:31,465 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:31,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:31,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:31,466 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:31,466 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:31,466 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:31,466 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,466 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:31,467 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:31,467 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:31,467 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,467 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:31,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:31,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:31,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:31,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:31,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:31,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:31,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:31,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:31,471 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:31,471 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:31,471 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:31,471 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:31,471 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:31,471 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:31,473 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:31,473 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:31,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:31,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:31,473 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:31,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:31,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:31,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:31,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:31,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:31,473 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:31,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:31,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:31,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:31,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:31,475 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:31,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:31,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,477 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,477 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:31,477 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:31,477 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:31,477 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:31,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:31,477 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:31,477 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:31,477 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:31,477 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:31,477 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:31,477 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:31,477 DEBUG [Thread-4140] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:31,477 DEBUG [Thread-4140] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:31,477 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:31,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:31,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:31,478 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:56:31,578 DEBUG [Thread-4140] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:31,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:31,579 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:31,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:31,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:31,581 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:31,581 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:31,581 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:31,581 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:31,581 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:31,581 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:31,583 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:31,583 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:31,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:31,584 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,584 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:31,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:31,584 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:31,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:31,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:31,584 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:31,585 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:31,585 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:31,585 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:31,585 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:31,585 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:31,585 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:31,585 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:31,585 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:31,585 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. started... 2023-03-31 16:56:31,585 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. as already flushing 2023-03-31 16:56:31,585 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:31,585 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:31,585 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:31,585 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,585 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:31,585 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:31,587 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:31,587 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:31,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:31,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:31,588 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:31,588 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:31,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:31,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:31,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:31,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:31,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11c9b31f[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:31,589 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:31,589 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:31,590 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:31,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:31,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:31,590 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:31,590 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:31,590 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:31,591 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,591 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:31,591 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:31,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:31,592 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:31,592 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:31,592 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:31,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:31,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:31,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:31,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:31,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:31,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:31,595 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:31,595 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:31,595 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:31,595 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:31,595 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:31,595 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:31,598 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:31,598 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:31,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:31,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:31,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:31,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:31,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:31,598 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:31,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:31,599 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:31,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:31,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:31,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:31,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:31,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:31,600 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:31,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:31,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,602 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:31,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:31,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:31,602 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:31,602 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:31,603 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:56:31,603 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:31,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:31,603 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:31,603 DEBUG [Thread-4140] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:31,603 DEBUG [Thread-4140] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:31,603 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:31,603 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,603 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:31,603 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:31,603 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:31,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:31,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:31,703 DEBUG [Thread-4140] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:31,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:31,705 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:31,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:31,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:31,707 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:31,707 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:31,707 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:31,707 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:31,707 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:31,707 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:31,709 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,709 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:31,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:31,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:31,709 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,709 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:31,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:31,710 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:31,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:31,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:31,710 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:31,710 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:31,710 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:31,710 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:31,711 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:31,711 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:31,711 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:31,711 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:31,711 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:31,711 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. started... 2023-03-31 16:56:31,711 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. as already flushing 2023-03-31 16:56:31,711 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:31,711 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:31,711 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:31,711 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,711 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:31,711 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:31,716 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,716 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:31,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:31,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:31,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:31,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:31,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:31,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:31,721 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:31,721 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65901c84[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:31,721 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:31,721 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:31,722 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,722 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:31,727 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:31,728 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:31,728 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:31,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:31,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:31,728 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:31,728 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:31,728 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:31,729 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:31,730 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:31,729 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,730 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:31,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:31,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:31,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:31,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:31,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:31,731 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:31,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49618 deadline: 1680281851728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:31,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:31,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:31,733 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:31,733 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:31,733 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:31,733 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:31,733 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:31,733 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:31,735 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:31,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:31,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:31,735 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:31,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:31,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:31,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:31,735 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:31,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:31,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:31,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:31,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:31,736 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:31,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:31,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:31,737 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:31,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:31,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,742 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:31,742 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,742 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:31,742 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:31,742 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,742 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:31,742 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:31,743 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:31,742 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:31,743 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:31,743 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:31,743 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:31,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:31,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:31,743 DEBUG [Thread-4140] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:31,743 DEBUG [Thread-4140] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:31,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:31,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:31,744 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:56:31,773 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:31,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49568 deadline: 1680281851771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,776 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:31,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49566 deadline: 1680281851775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,781 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:31,781 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-31 16:56:31,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49608 deadline: 1680281851777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33441] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49602 deadline: 1680281851778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,821 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=744 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/C/4a797b71e40a46aab4d04401cfd4d012 2023-03-31 16:56:31,825 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/A/53ee387229a547ecbaf01b7bce339d15 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/53ee387229a547ecbaf01b7bce339d15 2023-03-31 16:56:31,828 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/53ee387229a547ecbaf01b7bce339d15, entries=200, sequenceid=744, filesize=32.3 K 2023-03-31 16:56:31,829 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/B/1df5ff3f21244ada9d27c98c3d14937d as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/1df5ff3f21244ada9d27c98c3d14937d 2023-03-31 16:56:31,832 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/1df5ff3f21244ada9d27c98c3d14937d, entries=150, sequenceid=744, filesize=11.9 K 2023-03-31 16:56:31,833 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/C/4a797b71e40a46aab4d04401cfd4d012 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/4a797b71e40a46aab4d04401cfd4d012 2023-03-31 16:56:31,837 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/4a797b71e40a46aab4d04401cfd4d012, entries=150, sequenceid=744, filesize=11.9 K 2023-03-31 16:56:31,838 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~160.27 KB/164112, currentSize=127.47 KB/130530 for 313b3df11d64a9396698e7102a97394f in 1268ms, sequenceid=744, compaction requested=true 2023-03-31 16:56:31,838 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 313b3df11d64a9396698e7102a97394f: 2023-03-31 16:56:31,838 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:56:31,838 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 16:56:31,838 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-31 16:56:31,838 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:56:31,838 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:56:31,839 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37920 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 16:56:31,839 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 86540 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-31 16:56:31,839 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1912): 313b3df11d64a9396698e7102a97394f/B is initiating minor compaction (all files) 2023-03-31 16:56:31,839 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1912): 313b3df11d64a9396698e7102a97394f/A is initiating minor compaction (all files) 2023-03-31 16:56:31,839 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 313b3df11d64a9396698e7102a97394f/B in TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:31,839 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 313b3df11d64a9396698e7102a97394f/A in TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:31,839 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/81ce038b5a8e4456a5638120e711c97c, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/ef2d22bd65f84af09b5a784b002cc2fd, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/1df5ff3f21244ada9d27c98c3d14937d] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp, totalSize=37.0 K 2023-03-31 16:56:31,839 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/66b68c69223946c49982503f26de7a5e, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/4804b31b8abf48a09e72df1150feab1c, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/53ee387229a547ecbaf01b7bce339d15] into tmpdir=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp, totalSize=84.5 K 2023-03-31 16:56:31,840 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 81ce038b5a8e4456a5638120e711c97c, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=678, earliestPutTs=1680281788497 2023-03-31 16:56:31,840 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 66b68c69223946c49982503f26de7a5e, keycount=150, bloomtype=ROW, size=26.8 K, encoding=NONE, compression=NONE, seqNum=678, earliestPutTs=1680281788497 2023-03-31 16:56:31,840 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting ef2d22bd65f84af09b5a784b002cc2fd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=702, earliestPutTs=1680281790447 2023-03-31 16:56:31,840 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 4804b31b8abf48a09e72df1150feab1c, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=702, earliestPutTs=1680281790447 2023-03-31 16:56:31,840 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] compactions.Compactor(207): Compacting 1df5ff3f21244ada9d27c98c3d14937d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=744, earliestPutTs=1680281790562 2023-03-31 16:56:31,841 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.Compactor(207): Compacting 53ee387229a547ecbaf01b7bce339d15, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=744, earliestPutTs=1680281790560 2023-03-31 16:56:31,843 DEBUG [Thread-4140] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:31,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:31,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-31 16:56:31,847 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] throttle.PressureAwareThroughputController(145): 313b3df11d64a9396698e7102a97394f#B#compaction#689 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:56:31,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-31 16:56:31,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-31 16:56:31,851 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-31 16:56:31,851 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:31,851 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-31 16:56:31,851 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-31 16:56:31,852 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:31,852 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:31,853 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,853 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:31,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:31,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:31,854 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:31,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:31,854 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-31 16:56:31,854 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:31,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-31 16:56:31,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:31,854 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-31 16:56:31,856 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-31 16:56:31,856 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-31 16:56:31,856 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-31 16:56:31,856 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-31 16:56:31,856 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-31 16:56:31,871 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-31 16:56:31,872 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-31 16:56:31,872 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:31,872 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. started... 2023-03-31 16:56:31,872 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 313b3df11d64a9396698e7102a97394f 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-03-31 16:56:31,872 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=A 2023-03-31 16:56:31,872 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:56:31,872 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=B 2023-03-31 16:56:31,872 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:56:31,872 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=C 2023-03-31 16:56:31,872 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-31 16:56:31,873 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] throttle.PressureAwareThroughputController(145): 313b3df11d64a9396698e7102a97394f#A#compaction#690 average throughput is 1.16 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-31 16:56:31,908 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/A/76532879556145bcb630abe8a4ab7772 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/76532879556145bcb630abe8a4ab7772 2023-03-31 16:56:31,911 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 313b3df11d64a9396698e7102a97394f/A of 313b3df11d64a9396698e7102a97394f into 76532879556145bcb630abe8a4ab7772(size=26.9 K), total size for store is 26.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:56:31,912 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 313b3df11d64a9396698e7102a97394f: 2023-03-31 16:56:31,912 INFO [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f., storeName=313b3df11d64a9396698e7102a97394f/A, priority=13, startTime=1680281791838; duration=0sec 2023-03-31 16:56:31,912 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-31 16:56:31,912 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-03-31 16:56:31,912 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-31 16:56:31,913 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-31 16:56:31,913 DEBUG [RS:0;jenkins-hbase4:33441-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. because compaction request was cancelled 2023-03-31 16:56:32,064 DEBUG [Thread-4143] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4bb72437 to 127.0.0.1:51916 2023-03-31 16:56:32,064 DEBUG [Thread-4145] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3dca23c0 to 127.0.0.1:51916 2023-03-31 16:56:32,064 DEBUG [Thread-4143] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 16:56:32,064 DEBUG [Thread-4145] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 16:56:32,065 DEBUG [Thread-4141] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0a470a3d to 127.0.0.1:51916 2023-03-31 16:56:32,065 DEBUG [Thread-4141] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 16:56:32,065 DEBUG [Thread-4147] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x62d52321 to 127.0.0.1:51916 2023-03-31 16:56:32,065 DEBUG [Thread-4147] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 16:56:32,065 DEBUG [Thread-4149] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2a8bf279 to 127.0.0.1:51916 2023-03-31 16:56:32,066 DEBUG [Thread-4149] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 16:56:32,270 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/B/55e8d27ffb3f479cab27b67cfd4faada as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/55e8d27ffb3f479cab27b67cfd4faada 2023-03-31 16:56:32,273 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 313b3df11d64a9396698e7102a97394f/B of 313b3df11d64a9396698e7102a97394f into 55e8d27ffb3f479cab27b67cfd4faada(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-31 16:56:32,273 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 313b3df11d64a9396698e7102a97394f: 2023-03-31 16:56:32,273 INFO [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f., storeName=313b3df11d64a9396698e7102a97394f/B, priority=13, startTime=1680281791838; duration=0sec 2023-03-31 16:56:32,273 DEBUG [RS:0;jenkins-hbase4:33441-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-31 16:56:32,296 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230331a3844b78f0424f3a9c26864344bac01b to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331a3844b78f0424f3a9c26864344bac01b 2023-03-31 16:56:32,701 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=783, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/A/6b90aa9cbe124a7681ae0dfbb7d86a7b 2023-03-31 16:56:32,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33441] regionserver.HRegion(9158): Flush requested on 313b3df11d64a9396698e7102a97394f 2023-03-31 16:56:32,738 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. as already flushing 2023-03-31 16:56:32,738 DEBUG [Thread-4134] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x544a4215 to 127.0.0.1:51916 2023-03-31 16:56:32,738 DEBUG [Thread-4134] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 16:56:32,779 DEBUG [Thread-4132] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x66b5b1f3 to 127.0.0.1:51916 2023-03-31 16:56:32,779 DEBUG [Thread-4132] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 16:56:32,781 DEBUG [Thread-4130] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x062b3caa to 127.0.0.1:51916 2023-03-31 16:56:32,781 DEBUG [Thread-4130] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 16:56:32,791 DEBUG [Thread-4138] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2c53bf96 to 127.0.0.1:51916 2023-03-31 16:56:32,791 DEBUG [Thread-4138] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 16:56:32,792 DEBUG [Thread-4136] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x25baee3e to 127.0.0.1:51916 2023-03-31 16:56:32,792 DEBUG [Thread-4136] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 16:56:33,109 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=783 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/B/6d4ee07047c74f8bb40a2cfc831eae64 2023-03-31 16:56:33,517 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=783 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/C/dcb08d5b9f7d42979fce78ba672b334a 2023-03-31 16:56:33,520 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/A/6b90aa9cbe124a7681ae0dfbb7d86a7b as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/6b90aa9cbe124a7681ae0dfbb7d86a7b 2023-03-31 16:56:33,522 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/6b90aa9cbe124a7681ae0dfbb7d86a7b, entries=150, sequenceid=783, filesize=25.4 K 2023-03-31 16:56:33,523 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/B/6d4ee07047c74f8bb40a2cfc831eae64 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/6d4ee07047c74f8bb40a2cfc831eae64 2023-03-31 16:56:33,526 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/6d4ee07047c74f8bb40a2cfc831eae64, entries=150, sequenceid=783, filesize=11.9 K 2023-03-31 16:56:33,526 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/C/dcb08d5b9f7d42979fce78ba672b334a as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/dcb08d5b9f7d42979fce78ba672b334a 2023-03-31 16:56:33,529 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/dcb08d5b9f7d42979fce78ba672b334a, entries=150, sequenceid=783, filesize=11.9 K 2023-03-31 16:56:33,529 INFO [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=33.54 KB/34350 for 313b3df11d64a9396698e7102a97394f in 1657ms, sequenceid=783, compaction requested=true 2023-03-31 16:56:33,529 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 313b3df11d64a9396698e7102a97394f: 2023-03-31 16:56:33,529 DEBUG [rs(jenkins-hbase4.apache.org,33441,1680281619375)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:33,529 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-31 16:56:33,529 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-31 16:56:33,529 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:33,529 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-31 16:56:33,530 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-31 16:56:33,531 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:33,531 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:33,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:33,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:33,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:33,531 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:33,531 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-31 16:56:33,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:33,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:33,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:33,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:33,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:33,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33441,1680281619375' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-31 16:56:33,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ea46eb6[Count = 0] remaining members to acquire global barrier 2023-03-31 16:56:33,533 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-31 16:56:33,533 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:33,534 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:33,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:33,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:33,534 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-31 16:56:33,534 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-31 16:56:33,534 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,33441,1680281619375' in zk 2023-03-31 16:56:33,534 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:33,534 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-31 16:56:33,537 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:33,537 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-31 16:56:33,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:33,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:33,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:33,537 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:33,537 DEBUG [member: 'jenkins-hbase4.apache.org,33441,1680281619375' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-31 16:56:33,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:33,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:33,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:33,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:33,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:33,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:33,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:33,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,33441,1680281619375': 2023-03-31 16:56:33,539 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33441,1680281619375' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-31 16:56:33,539 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-31 16:56:33,539 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-31 16:56:33,539 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-31 16:56:33,539 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-31 16:56:33,539 INFO [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-31 16:56:33,541 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:33,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:33,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-31 16:56:33,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-31 16:56:33,541 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:33,541 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:33,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:33,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:33,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-31 16:56:33,541 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:33,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:33,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:33,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:33,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:33,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-31 16:56:33,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:33,542 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:33,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:33,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-31 16:56:33,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-31 16:56:33,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:33,544 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:33,544 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:33,544 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-31 16:56:33,544 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-31 16:56:33,544 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:33,544 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-31 16:56:33,544 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:33,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-31 16:56:33,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-31 16:56:33,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-31 16:56:33,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-31 16:56:33,544 DEBUG [(jenkins-hbase4.apache.org,43681,1680281617954)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-31 16:56:33,544 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-31 16:56:33,544 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-31 16:56:33,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-31 16:56:33,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-31 16:56:33,544 DEBUG [Thread-4140] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-31 16:56:33,544 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-31 16:56:33,544 DEBUG [Thread-4140] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-31 16:56:33,644 DEBUG [Thread-4140] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-31 16:56:33,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-31 16:56:33,645 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-31 16:56:33,646 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(394): wrote 144 2023-03-31 16:56:33,646 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(394): wrote 120 2023-03-31 16:56:33,646 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(394): wrote 140 2023-03-31 16:56:33,646 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(394): wrote 115 2023-03-31 16:56:33,646 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(394): wrote 126 2023-03-31 16:56:33,646 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-31 16:56:33,646 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(398): read 4053 2023-03-31 16:56:33,646 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(398): read 4054 2023-03-31 16:56:33,646 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(398): read 4058 2023-03-31 16:56:33,646 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(398): read 4074 2023-03-31 16:56:33,646 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(398): read 4063 2023-03-31 16:56:33,646 INFO [Listener at localhost/44135] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-31 16:56:33,646 INFO [Listener at localhost/44135] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-31 16:56:33,647 DEBUG [Listener at localhost/44135] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x59185ba7 to 127.0.0.1:51916 2023-03-31 16:56:33,647 DEBUG [Listener at localhost/44135] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 16:56:33,647 INFO [Listener at localhost/44135] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-31 16:56:33,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-03-31 16:56:33,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure2.ProcedureExecutor(1029): Stored pid=62, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-31 16:56:33,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-31 16:56:33,650 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680281793650"}]},"ts":"1680281793650"} 2023-03-31 16:56:33,651 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-31 16:56:33,653 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-31 16:56:33,653 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=313b3df11d64a9396698e7102a97394f, UNASSIGN}] 2023-03-31 16:56:33,655 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=313b3df11d64a9396698e7102a97394f, UNASSIGN 2023-03-31 16:56:33,655 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=313b3df11d64a9396698e7102a97394f, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:33,655 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680281793655"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680281793655"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680281793655"}]},"ts":"1680281793655"} 2023-03-31 16:56:33,656 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE; CloseRegionProcedure 313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375}] 2023-03-31 16:56:33,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-31 16:56:33,807 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 313b3df11d64a9396698e7102a97394f 2023-03-31 16:56:33,808 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 313b3df11d64a9396698e7102a97394f, disabling compactions & flushes 2023-03-31 16:56:33,808 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:33,808 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:33,808 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. after waiting 0 ms 2023-03-31 16:56:33,808 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:33,808 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 313b3df11d64a9396698e7102a97394f 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-03-31 16:56:33,808 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=A 2023-03-31 16:56:33,808 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:56:33,808 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=B 2023-03-31 16:56:33,808 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:56:33,808 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 313b3df11d64a9396698e7102a97394f, store=C 2023-03-31 16:56:33,808 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-31 16:56:33,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-31 16:56:34,219 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023033143273443709b4b658c743ef430de6997 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033143273443709b4b658c743ef430de6997 2023-03-31 16:56:34,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-31 16:56:34,623 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=793, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/A/12ef0fd9a4034359b6cd5dbef6877f10 2023-03-31 16:56:34,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-31 16:56:35,031 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=793 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/B/be3529856fe14f318ad0e00ee3692825 2023-03-31 16:56:35,442 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=793 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/C/4a1fb03f9d7b4aba98e74370e1ed4fc8 2023-03-31 16:56:35,445 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/A/12ef0fd9a4034359b6cd5dbef6877f10 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/12ef0fd9a4034359b6cd5dbef6877f10 2023-03-31 16:56:35,448 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/12ef0fd9a4034359b6cd5dbef6877f10, entries=150, sequenceid=793, filesize=25.4 K 2023-03-31 16:56:35,448 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/B/be3529856fe14f318ad0e00ee3692825 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/be3529856fe14f318ad0e00ee3692825 2023-03-31 16:56:35,451 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/be3529856fe14f318ad0e00ee3692825, entries=150, sequenceid=793, filesize=11.9 K 2023-03-31 16:56:35,452 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/.tmp/C/4a1fb03f9d7b4aba98e74370e1ed4fc8 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/4a1fb03f9d7b4aba98e74370e1ed4fc8 2023-03-31 16:56:35,455 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/4a1fb03f9d7b4aba98e74370e1ed4fc8, entries=150, sequenceid=793, filesize=11.9 K 2023-03-31 16:56:35,456 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for 313b3df11d64a9396698e7102a97394f in 1648ms, sequenceid=793, compaction requested=true 2023-03-31 16:56:35,479 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/eb5b7a0d6a874c09a9cbda570e57438e, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/d3673ed8fc5545708d85b0af48398a9f, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/39319352866046f294566564c5b8cdea, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/7c728df067244d8f86ca8fb8483f4c3d, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/36e4511ef13f47fcb823cbd9645b2920, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/6d962cc812da4b0181cc0215dff4d31b, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/41fc4ecadcd24322b94db4199aff434c, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/5e649225813a458d8a1636868b48bb1b, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/892b82e4bcde44d9a096cbaa9a7a846c, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/60d55b632371423ab562f8df426556d9, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/ee7a65ff8aac4e2baee210c3798c621c, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/2132eedb9a33436ca187dcc7974e49b3, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/af62b9f582dd4018897c68167000682c, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/18e62844b9d6410387523e7afef6d554, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/3410bb7b47364241a0b2f85cd2382136, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/a42fb449377441b0a1c54e26cc0af92b, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/04ce8f4061424fa487158be331c9e02d, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/dd1d4fe0b0b64966be099cb76c7aded0, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/5573c6fa554946628ea04eeaa68e9da2, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/da88ecdf5df8437f9cfa1f5ead0c9631, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/981fe047bd364ef384ac88e3d465f587, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/3ac01c86f9f94a6a8731c8c1ddd5c599, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/89cd51876936497d86c5513148efa8ec, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/cd8f70664250474e93489da2ecfb266e, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/76731a4c2ed14a328e1255343dd68c90, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/07c0dcfbd8384c0ea809fb8aebf5b3f4, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/d7aba367d4e94d7f83d53fc6d760d754, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/8c9817cf36734a5eb2019812038e2ee6, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/d342afa65b3943b5bd542ce5ad6be095, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/4a764573fca647129013af99f06ef144, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/202b0e2bcd154b65ac8a2459537deee1, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/0c573347799c4d70a899fdad86c1060a, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/700a7c97e96a4bff8157ca4bd033ce14, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/6dc56de547bb4e1b9b26a01c48ab1aae, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/08061d65a0a341e794ff5555c28719cf, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/1022e0ff14a94922adf2d183f0b3d968, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/59c210a68f5e44ca959bce675f425a8c, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/430bbd9a2cec43db942bfa4796567777, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/931939c421d443f3b847e296227ce9e5, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/66b68c69223946c49982503f26de7a5e, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/d3b73918b77448d1bbc2617c1921c17d, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/4804b31b8abf48a09e72df1150feab1c, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/53ee387229a547ecbaf01b7bce339d15] to archive 2023-03-31 16:56:35,479 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-31 16:56:35,481 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/eb5b7a0d6a874c09a9cbda570e57438e to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/eb5b7a0d6a874c09a9cbda570e57438e 2023-03-31 16:56:35,482 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/d3673ed8fc5545708d85b0af48398a9f to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/d3673ed8fc5545708d85b0af48398a9f 2023-03-31 16:56:35,483 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/39319352866046f294566564c5b8cdea to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/39319352866046f294566564c5b8cdea 2023-03-31 16:56:35,484 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/7c728df067244d8f86ca8fb8483f4c3d to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/7c728df067244d8f86ca8fb8483f4c3d 2023-03-31 16:56:35,485 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/36e4511ef13f47fcb823cbd9645b2920 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/36e4511ef13f47fcb823cbd9645b2920 2023-03-31 16:56:35,485 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/6d962cc812da4b0181cc0215dff4d31b to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/6d962cc812da4b0181cc0215dff4d31b 2023-03-31 16:56:35,486 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/41fc4ecadcd24322b94db4199aff434c to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/41fc4ecadcd24322b94db4199aff434c 2023-03-31 16:56:35,487 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/5e649225813a458d8a1636868b48bb1b to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/5e649225813a458d8a1636868b48bb1b 2023-03-31 16:56:35,488 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/892b82e4bcde44d9a096cbaa9a7a846c to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/892b82e4bcde44d9a096cbaa9a7a846c 2023-03-31 16:56:35,489 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/60d55b632371423ab562f8df426556d9 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/60d55b632371423ab562f8df426556d9 2023-03-31 16:56:35,490 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/ee7a65ff8aac4e2baee210c3798c621c to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/ee7a65ff8aac4e2baee210c3798c621c 2023-03-31 16:56:35,491 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/2132eedb9a33436ca187dcc7974e49b3 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/2132eedb9a33436ca187dcc7974e49b3 2023-03-31 16:56:35,491 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/af62b9f582dd4018897c68167000682c to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/af62b9f582dd4018897c68167000682c 2023-03-31 16:56:35,492 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/18e62844b9d6410387523e7afef6d554 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/18e62844b9d6410387523e7afef6d554 2023-03-31 16:56:35,493 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/3410bb7b47364241a0b2f85cd2382136 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/3410bb7b47364241a0b2f85cd2382136 2023-03-31 16:56:35,494 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/a42fb449377441b0a1c54e26cc0af92b to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/a42fb449377441b0a1c54e26cc0af92b 2023-03-31 16:56:35,495 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/04ce8f4061424fa487158be331c9e02d to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/04ce8f4061424fa487158be331c9e02d 2023-03-31 16:56:35,496 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/dd1d4fe0b0b64966be099cb76c7aded0 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/dd1d4fe0b0b64966be099cb76c7aded0 2023-03-31 16:56:35,497 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/5573c6fa554946628ea04eeaa68e9da2 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/5573c6fa554946628ea04eeaa68e9da2 2023-03-31 16:56:35,497 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/da88ecdf5df8437f9cfa1f5ead0c9631 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/da88ecdf5df8437f9cfa1f5ead0c9631 2023-03-31 16:56:35,498 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/981fe047bd364ef384ac88e3d465f587 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/981fe047bd364ef384ac88e3d465f587 2023-03-31 16:56:35,499 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/3ac01c86f9f94a6a8731c8c1ddd5c599 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/3ac01c86f9f94a6a8731c8c1ddd5c599 2023-03-31 16:56:35,500 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/89cd51876936497d86c5513148efa8ec to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/89cd51876936497d86c5513148efa8ec 2023-03-31 16:56:35,501 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/cd8f70664250474e93489da2ecfb266e to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/cd8f70664250474e93489da2ecfb266e 2023-03-31 16:56:35,502 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/76731a4c2ed14a328e1255343dd68c90 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/76731a4c2ed14a328e1255343dd68c90 2023-03-31 16:56:35,503 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/07c0dcfbd8384c0ea809fb8aebf5b3f4 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/07c0dcfbd8384c0ea809fb8aebf5b3f4 2023-03-31 16:56:35,503 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/d7aba367d4e94d7f83d53fc6d760d754 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/d7aba367d4e94d7f83d53fc6d760d754 2023-03-31 16:56:35,504 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/8c9817cf36734a5eb2019812038e2ee6 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/8c9817cf36734a5eb2019812038e2ee6 2023-03-31 16:56:35,505 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/d342afa65b3943b5bd542ce5ad6be095 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/d342afa65b3943b5bd542ce5ad6be095 2023-03-31 16:56:35,506 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/4a764573fca647129013af99f06ef144 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/4a764573fca647129013af99f06ef144 2023-03-31 16:56:35,507 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/202b0e2bcd154b65ac8a2459537deee1 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/202b0e2bcd154b65ac8a2459537deee1 2023-03-31 16:56:35,508 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/0c573347799c4d70a899fdad86c1060a to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/0c573347799c4d70a899fdad86c1060a 2023-03-31 16:56:35,509 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/700a7c97e96a4bff8157ca4bd033ce14 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/700a7c97e96a4bff8157ca4bd033ce14 2023-03-31 16:56:35,510 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/6dc56de547bb4e1b9b26a01c48ab1aae to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/6dc56de547bb4e1b9b26a01c48ab1aae 2023-03-31 16:56:35,511 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/08061d65a0a341e794ff5555c28719cf to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/08061d65a0a341e794ff5555c28719cf 2023-03-31 16:56:35,512 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/1022e0ff14a94922adf2d183f0b3d968 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/1022e0ff14a94922adf2d183f0b3d968 2023-03-31 16:56:35,513 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/59c210a68f5e44ca959bce675f425a8c to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/59c210a68f5e44ca959bce675f425a8c 2023-03-31 16:56:35,514 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/430bbd9a2cec43db942bfa4796567777 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/430bbd9a2cec43db942bfa4796567777 2023-03-31 16:56:35,515 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/931939c421d443f3b847e296227ce9e5 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/931939c421d443f3b847e296227ce9e5 2023-03-31 16:56:35,517 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/66b68c69223946c49982503f26de7a5e to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/66b68c69223946c49982503f26de7a5e 2023-03-31 16:56:35,518 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/d3b73918b77448d1bbc2617c1921c17d to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/d3b73918b77448d1bbc2617c1921c17d 2023-03-31 16:56:35,519 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/4804b31b8abf48a09e72df1150feab1c to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/4804b31b8abf48a09e72df1150feab1c 2023-03-31 16:56:35,520 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/53ee387229a547ecbaf01b7bce339d15 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/53ee387229a547ecbaf01b7bce339d15 2023-03-31 16:56:35,525 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/c59fa43c80e84d598d160426cd7f7e11, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/eb3aaf9091114180bdfb312b7b548ff5, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/58787737611d4d6fa2826a94387010f7, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/a962a4ad2d8f4dbcaf6fed629f966107, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/db0e267fa452488fae44c86b816db169, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/1df3c8a1e3a44325927604d51467e310, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/d249c4bb5e044bb2abaade955913e63d, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/f244f7e767144fcabbc899e4573c4ced, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/77318e9a09de4e2d8c7cbc694418ceb4, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/beaa85c63e094a9dbf109dd71b3a8d21, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/b253ed4b4dc84085a4474218345fd11d, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/a433bbefb54840d68735d3333436d8f0, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/37f223b1c19f4a2b8728c033ec88484a, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/1228fe929d5143a48e5835667413cce2, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/38084459697d4468be82f5c75bfd1779, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/5f1ff6b263554c819345d606c6ed8122, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/f4cc6521410c4a4ab11f6107ef25dd0c, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/afe5223c9fc442118374543638e2af09, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/beff2d13f7ad418e8cd8a51bfc0b119b, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/9bde24cd7f5f4177ac731f146292efa2, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/9aa80338984346b8832d4517ef6bca0a, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/63e9d6bf909a439f871cbffc37a5e9fb, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/bad83eefb17748ffa963902d4d3644d8, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/9a8f221abe2b4022a984ad62526e0b8f, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/5a7fa7b61d0945dbbd90c01655fba9b2, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/925e506e414b49d693cb635b7be2e8e8, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/2713f94bf7104e62828c8de3f3c1546e, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/e85d7be9433d42f785db9a9c0433ae6c, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/49a73bf8bd8e435dabc74cfa8581a782, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/28d2f2cc47a242b8b64f29a9a663b794, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/cbe85d56c0584bb198e635a9c3715ba1, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/f1fd3e829a03446e8ede1c55cbaec48a, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/d8d0a65ea0674c35bd4ffa6f3c4b4bc3, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/bfce5e92feec4a8ba92ed249fa70ec5f, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/81d9f303cbd14bdc9f2575b51e056ea9, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/fe49e2c7eadd43cda86e40eacec6c4bf, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/18ad2b903fb94f4892cd80601d154a44, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/68dfb52b2a264229be666c4a8e841a64, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/36340866c5b7411389d1e2b6045cecc5, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/81ce038b5a8e4456a5638120e711c97c, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/28151afcbff7410396fe650903aae418, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/ef2d22bd65f84af09b5a784b002cc2fd, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/1df5ff3f21244ada9d27c98c3d14937d] to archive 2023-03-31 16:56:35,525 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-31 16:56:35,527 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/c59fa43c80e84d598d160426cd7f7e11 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/c59fa43c80e84d598d160426cd7f7e11 2023-03-31 16:56:35,528 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/eb3aaf9091114180bdfb312b7b548ff5 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/eb3aaf9091114180bdfb312b7b548ff5 2023-03-31 16:56:35,529 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/58787737611d4d6fa2826a94387010f7 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/58787737611d4d6fa2826a94387010f7 2023-03-31 16:56:35,529 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/a962a4ad2d8f4dbcaf6fed629f966107 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/a962a4ad2d8f4dbcaf6fed629f966107 2023-03-31 16:56:35,530 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/db0e267fa452488fae44c86b816db169 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/db0e267fa452488fae44c86b816db169 2023-03-31 16:56:35,531 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/1df3c8a1e3a44325927604d51467e310 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/1df3c8a1e3a44325927604d51467e310 2023-03-31 16:56:35,532 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/d249c4bb5e044bb2abaade955913e63d to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/d249c4bb5e044bb2abaade955913e63d 2023-03-31 16:56:35,533 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/f244f7e767144fcabbc899e4573c4ced to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/f244f7e767144fcabbc899e4573c4ced 2023-03-31 16:56:35,533 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/77318e9a09de4e2d8c7cbc694418ceb4 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/77318e9a09de4e2d8c7cbc694418ceb4 2023-03-31 16:56:35,534 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/beaa85c63e094a9dbf109dd71b3a8d21 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/beaa85c63e094a9dbf109dd71b3a8d21 2023-03-31 16:56:35,535 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/b253ed4b4dc84085a4474218345fd11d to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/b253ed4b4dc84085a4474218345fd11d 2023-03-31 16:56:35,536 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/a433bbefb54840d68735d3333436d8f0 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/a433bbefb54840d68735d3333436d8f0 2023-03-31 16:56:35,536 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/37f223b1c19f4a2b8728c033ec88484a to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/37f223b1c19f4a2b8728c033ec88484a 2023-03-31 16:56:35,537 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/1228fe929d5143a48e5835667413cce2 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/1228fe929d5143a48e5835667413cce2 2023-03-31 16:56:35,538 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/38084459697d4468be82f5c75bfd1779 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/38084459697d4468be82f5c75bfd1779 2023-03-31 16:56:35,539 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/5f1ff6b263554c819345d606c6ed8122 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/5f1ff6b263554c819345d606c6ed8122 2023-03-31 16:56:35,540 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/f4cc6521410c4a4ab11f6107ef25dd0c to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/f4cc6521410c4a4ab11f6107ef25dd0c 2023-03-31 16:56:35,541 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/afe5223c9fc442118374543638e2af09 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/afe5223c9fc442118374543638e2af09 2023-03-31 16:56:35,542 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/beff2d13f7ad418e8cd8a51bfc0b119b to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/beff2d13f7ad418e8cd8a51bfc0b119b 2023-03-31 16:56:35,543 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/9bde24cd7f5f4177ac731f146292efa2 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/9bde24cd7f5f4177ac731f146292efa2 2023-03-31 16:56:35,544 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/9aa80338984346b8832d4517ef6bca0a to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/9aa80338984346b8832d4517ef6bca0a 2023-03-31 16:56:35,545 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/63e9d6bf909a439f871cbffc37a5e9fb to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/63e9d6bf909a439f871cbffc37a5e9fb 2023-03-31 16:56:35,546 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/bad83eefb17748ffa963902d4d3644d8 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/bad83eefb17748ffa963902d4d3644d8 2023-03-31 16:56:35,547 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/9a8f221abe2b4022a984ad62526e0b8f to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/9a8f221abe2b4022a984ad62526e0b8f 2023-03-31 16:56:35,548 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/5a7fa7b61d0945dbbd90c01655fba9b2 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/5a7fa7b61d0945dbbd90c01655fba9b2 2023-03-31 16:56:35,549 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/925e506e414b49d693cb635b7be2e8e8 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/925e506e414b49d693cb635b7be2e8e8 2023-03-31 16:56:35,550 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/2713f94bf7104e62828c8de3f3c1546e to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/2713f94bf7104e62828c8de3f3c1546e 2023-03-31 16:56:35,551 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/e85d7be9433d42f785db9a9c0433ae6c to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/e85d7be9433d42f785db9a9c0433ae6c 2023-03-31 16:56:35,552 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/49a73bf8bd8e435dabc74cfa8581a782 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/49a73bf8bd8e435dabc74cfa8581a782 2023-03-31 16:56:35,553 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/28d2f2cc47a242b8b64f29a9a663b794 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/28d2f2cc47a242b8b64f29a9a663b794 2023-03-31 16:56:35,554 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/cbe85d56c0584bb198e635a9c3715ba1 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/cbe85d56c0584bb198e635a9c3715ba1 2023-03-31 16:56:35,554 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/f1fd3e829a03446e8ede1c55cbaec48a to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/f1fd3e829a03446e8ede1c55cbaec48a 2023-03-31 16:56:35,555 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/d8d0a65ea0674c35bd4ffa6f3c4b4bc3 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/d8d0a65ea0674c35bd4ffa6f3c4b4bc3 2023-03-31 16:56:35,556 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/bfce5e92feec4a8ba92ed249fa70ec5f to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/bfce5e92feec4a8ba92ed249fa70ec5f 2023-03-31 16:56:35,557 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/81d9f303cbd14bdc9f2575b51e056ea9 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/81d9f303cbd14bdc9f2575b51e056ea9 2023-03-31 16:56:35,558 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/fe49e2c7eadd43cda86e40eacec6c4bf to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/fe49e2c7eadd43cda86e40eacec6c4bf 2023-03-31 16:56:35,558 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/18ad2b903fb94f4892cd80601d154a44 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/18ad2b903fb94f4892cd80601d154a44 2023-03-31 16:56:35,559 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/68dfb52b2a264229be666c4a8e841a64 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/68dfb52b2a264229be666c4a8e841a64 2023-03-31 16:56:35,560 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/36340866c5b7411389d1e2b6045cecc5 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/36340866c5b7411389d1e2b6045cecc5 2023-03-31 16:56:35,561 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/81ce038b5a8e4456a5638120e711c97c to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/81ce038b5a8e4456a5638120e711c97c 2023-03-31 16:56:35,562 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/28151afcbff7410396fe650903aae418 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/28151afcbff7410396fe650903aae418 2023-03-31 16:56:35,562 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/ef2d22bd65f84af09b5a784b002cc2fd to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/ef2d22bd65f84af09b5a784b002cc2fd 2023-03-31 16:56:35,563 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/1df5ff3f21244ada9d27c98c3d14937d to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/1df5ff3f21244ada9d27c98c3d14937d 2023-03-31 16:56:35,566 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/5a8ff4c9182740e6927a7a1dd5b32dde, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/ec02caf3bdaf4027a5bd1ab53d2d9c0c, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/dd27abe1f1c942da90188f1f69ba5ee2, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/f93eff84a48f4b20ab9b211d2a010793, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/5f9c96f314314f298418c829edf13340, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/029814b4ca664c32bc1461d3a3d075ca, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/ca066de47a0d41f182a2ba3e703caa04, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/60babfe39e684c5c9af1b356d6a6b69d, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/c9844faef41c4de88e6af9777b711d5b, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/6694aed74b3949ed872eba616ce0665c, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/02c00cbf15df4f7ab7fc69d129eef442, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/435cc93dd6194ef2bc78a80c47d8af90, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/dac5c4273b9e4ef8a74f0382cda66cff, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/d4ff7a2d08564fd09d62080417fee920, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/d36534d94d3247f7a39e082d59684576, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/fcf6fe37d91e4ac49edfeb7d1761ee77, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/78344cc17bda41639d06cbd5c7f81eec, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/9ca45eb906334f76b38464b9659bc024, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/9b743e44bca748e19b9090ccd77cacbb, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/2ed2ba619a2045a8985ee92ac6f6f816, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/915e0411902b4a238f2fc3675d32367b, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/fdae65420483445f92bc922c07c09ffa, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/20feb655ad96480b858e0481fb790904, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/3bbd80c48d7b4a91a7c50c82965ba696, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/c3928f2ef8794f908165b2f1b50dffb1, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/9480f34e9a3141c4acfb9dd1fea570a1, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/34cf8bf76a654dcab805588a759cc936, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/a3956fabe99b4d429b02a4b8c8634937, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/2d5e1c6bee084c3193507c9821cb2ec2, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/8c21ee7f2d1d42c99a6f0495e40b65ce, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/4cfccbbe58cb42c8897cde42382679b4, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/f691f015cb3a43c3bf17fd7ca7b13f0f, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/98adfa9865b9481fb1da72363f3419cb, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/6dd9689ef20e41e08bc9f56a4c194358, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/3a4668a0b2ab46029670fbf3e8649965, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/e5dbf7fcfcba48d2af9a88d96a923aac, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/ac262b67896b470491838238721d0b5d, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/4c1f9ca92df04bf3b1a7c3153dee20aa, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/cd1bbcd079304c228ff1a883b3d220b6, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/b263025817544911894ba37ce2d9f0c5, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/8b7a3c94058a4ba09c03af8d5ae409f3] to archive 2023-03-31 16:56:35,567 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-31 16:56:35,569 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/5a8ff4c9182740e6927a7a1dd5b32dde to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/5a8ff4c9182740e6927a7a1dd5b32dde 2023-03-31 16:56:35,570 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/ec02caf3bdaf4027a5bd1ab53d2d9c0c to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/ec02caf3bdaf4027a5bd1ab53d2d9c0c 2023-03-31 16:56:35,571 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/dd27abe1f1c942da90188f1f69ba5ee2 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/dd27abe1f1c942da90188f1f69ba5ee2 2023-03-31 16:56:35,572 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/f93eff84a48f4b20ab9b211d2a010793 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/f93eff84a48f4b20ab9b211d2a010793 2023-03-31 16:56:35,573 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/5f9c96f314314f298418c829edf13340 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/5f9c96f314314f298418c829edf13340 2023-03-31 16:56:35,575 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/029814b4ca664c32bc1461d3a3d075ca to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/029814b4ca664c32bc1461d3a3d075ca 2023-03-31 16:56:35,576 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/ca066de47a0d41f182a2ba3e703caa04 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/ca066de47a0d41f182a2ba3e703caa04 2023-03-31 16:56:35,577 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/60babfe39e684c5c9af1b356d6a6b69d to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/60babfe39e684c5c9af1b356d6a6b69d 2023-03-31 16:56:35,578 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/c9844faef41c4de88e6af9777b711d5b to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/c9844faef41c4de88e6af9777b711d5b 2023-03-31 16:56:35,579 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/6694aed74b3949ed872eba616ce0665c to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/6694aed74b3949ed872eba616ce0665c 2023-03-31 16:56:35,580 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/02c00cbf15df4f7ab7fc69d129eef442 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/02c00cbf15df4f7ab7fc69d129eef442 2023-03-31 16:56:35,581 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/435cc93dd6194ef2bc78a80c47d8af90 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/435cc93dd6194ef2bc78a80c47d8af90 2023-03-31 16:56:35,582 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/dac5c4273b9e4ef8a74f0382cda66cff to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/dac5c4273b9e4ef8a74f0382cda66cff 2023-03-31 16:56:35,583 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/d4ff7a2d08564fd09d62080417fee920 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/d4ff7a2d08564fd09d62080417fee920 2023-03-31 16:56:35,584 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/d36534d94d3247f7a39e082d59684576 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/d36534d94d3247f7a39e082d59684576 2023-03-31 16:56:35,585 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/fcf6fe37d91e4ac49edfeb7d1761ee77 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/fcf6fe37d91e4ac49edfeb7d1761ee77 2023-03-31 16:56:35,586 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/78344cc17bda41639d06cbd5c7f81eec to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/78344cc17bda41639d06cbd5c7f81eec 2023-03-31 16:56:35,587 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/9ca45eb906334f76b38464b9659bc024 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/9ca45eb906334f76b38464b9659bc024 2023-03-31 16:56:35,589 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/9b743e44bca748e19b9090ccd77cacbb to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/9b743e44bca748e19b9090ccd77cacbb 2023-03-31 16:56:35,590 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/2ed2ba619a2045a8985ee92ac6f6f816 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/2ed2ba619a2045a8985ee92ac6f6f816 2023-03-31 16:56:35,591 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/915e0411902b4a238f2fc3675d32367b to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/915e0411902b4a238f2fc3675d32367b 2023-03-31 16:56:35,591 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/fdae65420483445f92bc922c07c09ffa to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/fdae65420483445f92bc922c07c09ffa 2023-03-31 16:56:35,592 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/20feb655ad96480b858e0481fb790904 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/20feb655ad96480b858e0481fb790904 2023-03-31 16:56:35,593 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/3bbd80c48d7b4a91a7c50c82965ba696 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/3bbd80c48d7b4a91a7c50c82965ba696 2023-03-31 16:56:35,594 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/c3928f2ef8794f908165b2f1b50dffb1 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/c3928f2ef8794f908165b2f1b50dffb1 2023-03-31 16:56:35,595 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/9480f34e9a3141c4acfb9dd1fea570a1 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/9480f34e9a3141c4acfb9dd1fea570a1 2023-03-31 16:56:35,596 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/34cf8bf76a654dcab805588a759cc936 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/34cf8bf76a654dcab805588a759cc936 2023-03-31 16:56:35,597 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/a3956fabe99b4d429b02a4b8c8634937 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/a3956fabe99b4d429b02a4b8c8634937 2023-03-31 16:56:35,598 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/2d5e1c6bee084c3193507c9821cb2ec2 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/2d5e1c6bee084c3193507c9821cb2ec2 2023-03-31 16:56:35,599 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/8c21ee7f2d1d42c99a6f0495e40b65ce to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/8c21ee7f2d1d42c99a6f0495e40b65ce 2023-03-31 16:56:35,600 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/4cfccbbe58cb42c8897cde42382679b4 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/4cfccbbe58cb42c8897cde42382679b4 2023-03-31 16:56:35,601 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/f691f015cb3a43c3bf17fd7ca7b13f0f to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/f691f015cb3a43c3bf17fd7ca7b13f0f 2023-03-31 16:56:35,603 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/98adfa9865b9481fb1da72363f3419cb to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/98adfa9865b9481fb1da72363f3419cb 2023-03-31 16:56:35,604 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/6dd9689ef20e41e08bc9f56a4c194358 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/6dd9689ef20e41e08bc9f56a4c194358 2023-03-31 16:56:35,605 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/3a4668a0b2ab46029670fbf3e8649965 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/3a4668a0b2ab46029670fbf3e8649965 2023-03-31 16:56:35,606 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/e5dbf7fcfcba48d2af9a88d96a923aac to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/e5dbf7fcfcba48d2af9a88d96a923aac 2023-03-31 16:56:35,607 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/ac262b67896b470491838238721d0b5d to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/ac262b67896b470491838238721d0b5d 2023-03-31 16:56:35,608 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/4c1f9ca92df04bf3b1a7c3153dee20aa to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/4c1f9ca92df04bf3b1a7c3153dee20aa 2023-03-31 16:56:35,609 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/cd1bbcd079304c228ff1a883b3d220b6 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/cd1bbcd079304c228ff1a883b3d220b6 2023-03-31 16:56:35,610 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/b263025817544911894ba37ce2d9f0c5 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/b263025817544911894ba37ce2d9f0c5 2023-03-31 16:56:35,610 DEBUG [StoreCloser-TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/8b7a3c94058a4ba09c03af8d5ae409f3 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/8b7a3c94058a4ba09c03af8d5ae409f3 2023-03-31 16:56:35,614 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/recovered.edits/796.seqid, newMaxSeqId=796, maxSeqId=4 2023-03-31 16:56:35,616 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f. 2023-03-31 16:56:35,616 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 313b3df11d64a9396698e7102a97394f: 2023-03-31 16:56:35,617 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 313b3df11d64a9396698e7102a97394f 2023-03-31 16:56:35,617 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=313b3df11d64a9396698e7102a97394f, regionState=CLOSED 2023-03-31 16:56:35,617 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680281795617"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680281795617"}]},"ts":"1680281795617"} 2023-03-31 16:56:35,619 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=64, resume processing ppid=63 2023-03-31 16:56:35,619 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; CloseRegionProcedure 313b3df11d64a9396698e7102a97394f, server=jenkins-hbase4.apache.org,33441,1680281619375 in 1.9620 sec 2023-03-31 16:56:35,619 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=63, resume processing ppid=62 2023-03-31 16:56:35,619 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=63, ppid=62, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=313b3df11d64a9396698e7102a97394f, UNASSIGN in 1.9660 sec 2023-03-31 16:56:35,620 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680281795620"}]},"ts":"1680281795620"} 2023-03-31 16:56:35,620 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-31 16:56:35,622 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-31 16:56:35,623 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=62, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.9740 sec 2023-03-31 16:56:35,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-31 16:56:35,755 INFO [Listener at localhost/44135] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 62 completed 2023-03-31 16:56:35,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-03-31 16:56:35,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] procedure2.ProcedureExecutor(1029): Stored pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-31 16:56:35,757 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-31 16:56:35,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-03-31 16:56:35,758 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=65, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-31 16:56:35,763 DEBUG [HFileArchiver-13] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f 2023-03-31 16:56:35,765 DEBUG [HFileArchiver-13] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A, FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B, FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C, FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/recovered.edits] 2023-03-31 16:56:35,767 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/12ef0fd9a4034359b6cd5dbef6877f10 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/12ef0fd9a4034359b6cd5dbef6877f10 2023-03-31 16:56:35,767 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/6b90aa9cbe124a7681ae0dfbb7d86a7b to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/6b90aa9cbe124a7681ae0dfbb7d86a7b 2023-03-31 16:56:35,768 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/76532879556145bcb630abe8a4ab7772 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/A/76532879556145bcb630abe8a4ab7772 2023-03-31 16:56:35,770 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/55e8d27ffb3f479cab27b67cfd4faada to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/55e8d27ffb3f479cab27b67cfd4faada 2023-03-31 16:56:35,771 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/6d4ee07047c74f8bb40a2cfc831eae64 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/6d4ee07047c74f8bb40a2cfc831eae64 2023-03-31 16:56:35,772 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/be3529856fe14f318ad0e00ee3692825 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/B/be3529856fe14f318ad0e00ee3692825 2023-03-31 16:56:35,774 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/00ce0de93a254e8fac8937297e525720 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/00ce0de93a254e8fac8937297e525720 2023-03-31 16:56:35,775 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/4a1fb03f9d7b4aba98e74370e1ed4fc8 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/4a1fb03f9d7b4aba98e74370e1ed4fc8 2023-03-31 16:56:35,775 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/4a797b71e40a46aab4d04401cfd4d012 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/4a797b71e40a46aab4d04401cfd4d012 2023-03-31 16:56:35,776 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/dcb08d5b9f7d42979fce78ba672b334a to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/C/dcb08d5b9f7d42979fce78ba672b334a 2023-03-31 16:56:35,778 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/recovered.edits/796.seqid to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f/recovered.edits/796.seqid 2023-03-31 16:56:35,779 DEBUG [HFileArchiver-13] backup.HFileArchiver(596): Deleted hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/.tmp/data/default/TestAcidGuarantees/313b3df11d64a9396698e7102a97394f 2023-03-31 16:56:35,779 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-31 16:56:35,779 DEBUG [PEWorker-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-31 16:56:35,779 DEBUG [PEWorker-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-03-31 16:56:35,782 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303310d1b991189384d0290a4d4a84ad420f4 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303310d1b991189384d0290a4d4a84ad420f4 2023-03-31 16:56:35,782 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303310db1873573794e1b9967bba088b428a1 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303310db1873573794e1b9967bba088b428a1 2023-03-31 16:56:35,783 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303310eb8f78746064daf855da8567d296539 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303310eb8f78746064daf855da8567d296539 2023-03-31 16:56:35,784 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033117507ce9012548ad877156ac46d97008 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033117507ce9012548ad877156ac46d97008 2023-03-31 16:56:35,785 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303312442210389ce480eb1e3928489a2a2c2 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303312442210389ce480eb1e3928489a2a2c2 2023-03-31 16:56:35,786 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303313609ae91b80249cebf4ba29f4e95a6a2 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303313609ae91b80249cebf4ba29f4e95a6a2 2023-03-31 16:56:35,787 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303313a1dec17291c423d97381321f86594b2 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303313a1dec17291c423d97381321f86594b2 2023-03-31 16:56:35,788 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303313cc852c6456f4e8794949816c9ca3f5a to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303313cc852c6456f4e8794949816c9ca3f5a 2023-03-31 16:56:35,789 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303313f7eb0825229442a8e787e5449b3a507 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303313f7eb0825229442a8e787e5449b3a507 2023-03-31 16:56:35,790 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033142808fd488d944aa9d66f761e64d259c to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033142808fd488d944aa9d66f761e64d259c 2023-03-31 16:56:35,791 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033143273443709b4b658c743ef430de6997 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033143273443709b4b658c743ef430de6997 2023-03-31 16:56:35,791 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331589fa342861545e8b23693b4c8d931b7 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331589fa342861545e8b23693b4c8d931b7 2023-03-31 16:56:35,792 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303315c0714100447410fa4a23d82252a2a34 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303315c0714100447410fa4a23d82252a2a34 2023-03-31 16:56:35,793 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033163e9cc7123544744b4014236d602e63e to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033163e9cc7123544744b4014236d602e63e 2023-03-31 16:56:35,794 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033164a3e82d210e4020bb6b10269707ca4b to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033164a3e82d210e4020bb6b10269707ca4b 2023-03-31 16:56:35,795 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331694841752f9d4c9c8a1f4e49c8ccc251 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331694841752f9d4c9c8a1f4e49c8ccc251 2023-03-31 16:56:35,795 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033171ba1280c2864d7fbdd1ab775c218805 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033171ba1280c2864d7fbdd1ab775c218805 2023-03-31 16:56:35,796 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303317555983aa23247349385e7a44a6a921f to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303317555983aa23247349385e7a44a6a921f 2023-03-31 16:56:35,797 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303317702cc5413a0447d95c9eac72b2d0cb5 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303317702cc5413a0447d95c9eac72b2d0cb5 2023-03-31 16:56:35,798 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303317bf20aa0ba424b2eb7115ac10432dd7f to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303317bf20aa0ba424b2eb7115ac10432dd7f 2023-03-31 16:56:35,799 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033184aa2f97bcc54b978542188fbf4b0444 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033184aa2f97bcc54b978542188fbf4b0444 2023-03-31 16:56:35,800 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331883da464badd4f29ad3ef30f998a720d to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331883da464badd4f29ad3ef30f998a720d 2023-03-31 16:56:35,801 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033195b93d84fe6740f89676408aee92e41f to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033195b93d84fe6740f89676408aee92e41f 2023-03-31 16:56:35,802 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303319ca1fa3ff55f462e94dcf6230fc23a15 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303319ca1fa3ff55f462e94dcf6230fc23a15 2023-03-31 16:56:35,803 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303319e8a55c96dd04286ac791daad77ad965 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303319e8a55c96dd04286ac791daad77ad965 2023-03-31 16:56:35,804 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331a3844b78f0424f3a9c26864344bac01b to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331a3844b78f0424f3a9c26864344bac01b 2023-03-31 16:56:35,805 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331a7896db82e8d4e71ade75f0f1de5e877 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331a7896db82e8d4e71ade75f0f1de5e877 2023-03-31 16:56:35,806 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331aea0ee43858243deadbdd38525c61019 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331aea0ee43858243deadbdd38525c61019 2023-03-31 16:56:35,807 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331b52882b9355b474196a2371681394258 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331b52882b9355b474196a2371681394258 2023-03-31 16:56:35,808 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331c5242b245190417baf1d747a2d95b78c to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331c5242b245190417baf1d747a2d95b78c 2023-03-31 16:56:35,809 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331c95713d984a449f0b7231e0297e465ec to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331c95713d984a449f0b7231e0297e465ec 2023-03-31 16:56:35,810 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331cddbbfd90355459e9f5c1ad73be4ec23 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331cddbbfd90355459e9f5c1ad73be4ec23 2023-03-31 16:56:35,810 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331d550ba2a1f764528855d8f9815bdab11 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331d550ba2a1f764528855d8f9815bdab11 2023-03-31 16:56:35,811 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331dbfb39a2a71948e7bc824d45664bb5c9 to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331dbfb39a2a71948e7bc824d45664bb5c9 2023-03-31 16:56:35,812 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331e7f3708c696b4bb39dc2afa84b1b16ce to hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230331e7f3708c696b4bb39dc2afa84b1b16ce 2023-03-31 16:56:35,813 DEBUG [PEWorker-5] backup.HFileArchiver(596): Deleted hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-31 16:56:35,815 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=65, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-31 16:56:35,816 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-31 16:56:35,817 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-31 16:56:35,818 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=65, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-31 16:56:35,818 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-31 16:56:35,818 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680281795818"}]},"ts":"9223372036854775807"} 2023-03-31 16:56:35,819 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-31 16:56:35,819 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 313b3df11d64a9396698e7102a97394f, NAME => 'TestAcidGuarantees,,1680281768964.313b3df11d64a9396698e7102a97394f.', STARTKEY => '', ENDKEY => ''}] 2023-03-31 16:56:35,819 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-31 16:56:35,820 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680281795820"}]},"ts":"9223372036854775807"} 2023-03-31 16:56:35,821 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-31 16:56:35,822 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=65, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-31 16:56:35,823 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=65, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 67 msec 2023-03-31 16:56:35,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43681] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-03-31 16:56:35,859 INFO [Listener at localhost/44135] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 65 completed 2023-03-31 16:56:35,870 INFO [Listener at localhost/44135] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=228 (was 229), OpenFileDescriptor=566 (was 564) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=474 (was 403) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=3993 (was 3951) - AvailableMemoryMB LEAK? - 2023-03-31 16:56:35,870 INFO [Listener at localhost/44135] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-03-31 16:56:35,871 INFO [Listener at localhost/44135] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-31 16:56:35,871 DEBUG [Listener at localhost/44135] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6ed65498 to 127.0.0.1:51916 2023-03-31 16:56:35,871 DEBUG [Listener at localhost/44135] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 16:56:35,871 DEBUG [Listener at localhost/44135] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-03-31 16:56:35,871 DEBUG [Listener at localhost/44135] util.JVMClusterUtil(257): Found active master hash=543680963, stopped=false 2023-03-31 16:56:35,871 INFO [Listener at localhost/44135] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,43681,1680281617954 2023-03-31 16:56:35,872 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-03-31 16:56:35,873 INFO [Listener at localhost/44135] procedure2.ProcedureExecutor(629): Stopping 2023-03-31 16:56:35,873 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-31 16:56:35,873 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-03-31 16:56:35,873 DEBUG [Listener at localhost/44135] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7c909e1b to 127.0.0.1:51916 2023-03-31 16:56:35,873 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-03-31 16:56:35,873 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-03-31 16:56:35,873 DEBUG [Listener at localhost/44135] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 16:56:35,873 INFO [Listener at localhost/44135] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,33441,1680281619375' ***** 2023-03-31 16:56:35,874 INFO [Listener at localhost/44135] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-03-31 16:56:35,874 INFO [RS:0;jenkins-hbase4:33441] regionserver.HeapMemoryManager(220): Stopping 2023-03-31 16:56:35,874 INFO [RS:0;jenkins-hbase4:33441] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-03-31 16:56:35,874 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-03-31 16:56:35,874 INFO [RS:0;jenkins-hbase4:33441] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-03-31 16:56:35,874 INFO [RS:0;jenkins-hbase4:33441] regionserver.HRegionServer(3304): Received CLOSE for 4920086c0c3253e0dccccde39038f1cc 2023-03-31 16:56:35,875 INFO [RS:0;jenkins-hbase4:33441] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:35,875 DEBUG [RS:0;jenkins-hbase4:33441] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7a67ad5b to 127.0.0.1:51916 2023-03-31 16:56:35,876 DEBUG [RS:0;jenkins-hbase4:33441] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 16:56:35,876 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 4920086c0c3253e0dccccde39038f1cc, disabling compactions & flushes 2023-03-31 16:56:35,876 INFO [RS:0;jenkins-hbase4:33441] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-03-31 16:56:35,876 INFO [RS:0;jenkins-hbase4:33441] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-03-31 16:56:35,876 INFO [RS:0;jenkins-hbase4:33441] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-03-31 16:56:35,876 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1680281622054.4920086c0c3253e0dccccde39038f1cc. 2023-03-31 16:56:35,876 INFO [RS:0;jenkins-hbase4:33441] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-03-31 16:56:35,876 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1680281622054.4920086c0c3253e0dccccde39038f1cc. 2023-03-31 16:56:35,876 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1680281622054.4920086c0c3253e0dccccde39038f1cc. after waiting 0 ms 2023-03-31 16:56:35,876 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1680281622054.4920086c0c3253e0dccccde39038f1cc. 2023-03-31 16:56:35,876 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 4920086c0c3253e0dccccde39038f1cc 1/1 column families, dataSize=78 B heapSize=488 B 2023-03-31 16:56:35,876 INFO [RS:0;jenkins-hbase4:33441] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-03-31 16:56:35,876 DEBUG [RS:0;jenkins-hbase4:33441] regionserver.HRegionServer(1479): Online Regions={1588230740=hbase:meta,,1.1588230740, 4920086c0c3253e0dccccde39038f1cc=hbase:namespace,,1680281622054.4920086c0c3253e0dccccde39038f1cc.} 2023-03-31 16:56:35,877 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-03-31 16:56:35,877 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-03-31 16:56:35,877 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-03-31 16:56:35,877 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-03-31 16:56:35,877 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-03-31 16:56:35,877 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=20.92 KB heapSize=36.24 KB 2023-03-31 16:56:35,878 DEBUG [RS:0;jenkins-hbase4:33441] regionserver.HRegionServer(1505): Waiting on 1588230740, 4920086c0c3253e0dccccde39038f1cc 2023-03-31 16:56:35,917 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-03-31 16:56:36,079 DEBUG [RS:0;jenkins-hbase4:33441] regionserver.HRegionServer(1505): Waiting on 1588230740, 4920086c0c3253e0dccccde39038f1cc 2023-03-31 16:56:36,279 DEBUG [RS:0;jenkins-hbase4:33441] regionserver.HRegionServer(1505): Waiting on 1588230740, 4920086c0c3253e0dccccde39038f1cc 2023-03-31 16:56:36,283 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/hbase/namespace/4920086c0c3253e0dccccde39038f1cc/.tmp/info/75ebb0797d9f4347ab5c1172149c2537 2023-03-31 16:56:36,283 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=18.26 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/hbase/meta/1588230740/.tmp/info/cc3303c05aba419ab8d697812aebed81 2023-03-31 16:56:36,288 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/hbase/namespace/4920086c0c3253e0dccccde39038f1cc/.tmp/info/75ebb0797d9f4347ab5c1172149c2537 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/hbase/namespace/4920086c0c3253e0dccccde39038f1cc/info/75ebb0797d9f4347ab5c1172149c2537 2023-03-31 16:56:36,288 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for cc3303c05aba419ab8d697812aebed81 2023-03-31 16:56:36,291 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/hbase/namespace/4920086c0c3253e0dccccde39038f1cc/info/75ebb0797d9f4347ab5c1172149c2537, entries=2, sequenceid=6, filesize=4.8 K 2023-03-31 16:56:36,292 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 4920086c0c3253e0dccccde39038f1cc in 416ms, sequenceid=6, compaction requested=false 2023-03-31 16:56:36,298 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/hbase/namespace/4920086c0c3253e0dccccde39038f1cc/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-03-31 16:56:36,299 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1680281622054.4920086c0c3253e0dccccde39038f1cc. 2023-03-31 16:56:36,299 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 4920086c0c3253e0dccccde39038f1cc: 2023-03-31 16:56:36,299 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1680281622054.4920086c0c3253e0dccccde39038f1cc. 2023-03-31 16:56:36,479 DEBUG [RS:0;jenkins-hbase4:33441] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-03-31 16:56:36,679 DEBUG [RS:0;jenkins-hbase4:33441] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-03-31 16:56:36,685 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-03-31 16:56:36,685 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-03-31 16:56:36,696 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=588 B at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/hbase/meta/1588230740/.tmp/rep_barrier/656c7826ac29473ebbdf58ea10051752 2023-03-31 16:56:36,699 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 656c7826ac29473ebbdf58ea10051752 2023-03-31 16:56:36,880 INFO [RS:0;jenkins-hbase4:33441] regionserver.HRegionServer(1475): Waiting on 1 regions to close 2023-03-31 16:56:36,880 DEBUG [RS:0;jenkins-hbase4:33441] regionserver.HRegionServer(1479): Online Regions={1588230740=hbase:meta,,1.1588230740} 2023-03-31 16:56:36,880 DEBUG [RS:0;jenkins-hbase4:33441] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-03-31 16:56:37,080 DEBUG [RS:0;jenkins-hbase4:33441] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-03-31 16:56:37,104 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.08 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/hbase/meta/1588230740/.tmp/table/829678cb2a294319a22594e01f05baea 2023-03-31 16:56:37,107 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 829678cb2a294319a22594e01f05baea 2023-03-31 16:56:37,108 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/hbase/meta/1588230740/.tmp/info/cc3303c05aba419ab8d697812aebed81 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/hbase/meta/1588230740/info/cc3303c05aba419ab8d697812aebed81 2023-03-31 16:56:37,111 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for cc3303c05aba419ab8d697812aebed81 2023-03-31 16:56:37,111 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/hbase/meta/1588230740/info/cc3303c05aba419ab8d697812aebed81, entries=22, sequenceid=93, filesize=7.1 K 2023-03-31 16:56:37,112 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/hbase/meta/1588230740/.tmp/rep_barrier/656c7826ac29473ebbdf58ea10051752 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/hbase/meta/1588230740/rep_barrier/656c7826ac29473ebbdf58ea10051752 2023-03-31 16:56:37,115 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 656c7826ac29473ebbdf58ea10051752 2023-03-31 16:56:37,115 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/hbase/meta/1588230740/rep_barrier/656c7826ac29473ebbdf58ea10051752, entries=6, sequenceid=93, filesize=5.5 K 2023-03-31 16:56:37,115 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/hbase/meta/1588230740/.tmp/table/829678cb2a294319a22594e01f05baea as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/hbase/meta/1588230740/table/829678cb2a294319a22594e01f05baea 2023-03-31 16:56:37,118 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 829678cb2a294319a22594e01f05baea 2023-03-31 16:56:37,118 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/hbase/meta/1588230740/table/829678cb2a294319a22594e01f05baea, entries=9, sequenceid=93, filesize=5.5 K 2023-03-31 16:56:37,118 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.92 KB/21417, heapSize ~36.20 KB/37064, currentSize=0 B/0 for 1588230740 in 1241ms, sequenceid=93, compaction requested=false 2023-03-31 16:56:37,126 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/data/hbase/meta/1588230740/recovered.edits/96.seqid, newMaxSeqId=96, maxSeqId=1 2023-03-31 16:56:37,127 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-03-31 16:56:37,128 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-03-31 16:56:37,128 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-03-31 16:56:37,128 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-03-31 16:56:37,280 INFO [RS:0;jenkins-hbase4:33441] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,33441,1680281619375; all regions closed. 2023-03-31 16:56:37,288 DEBUG [RS:0;jenkins-hbase4:33441] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/oldWALs 2023-03-31 16:56:37,288 INFO [RS:0;jenkins-hbase4:33441] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C33441%2C1680281619375.meta:.meta(num 1680281621813) 2023-03-31 16:56:37,292 DEBUG [RS:0;jenkins-hbase4:33441] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/oldWALs 2023-03-31 16:56:37,293 INFO [RS:0;jenkins-hbase4:33441] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C33441%2C1680281619375:(num 1680281620838) 2023-03-31 16:56:37,293 DEBUG [RS:0;jenkins-hbase4:33441] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 16:56:37,293 INFO [RS:0;jenkins-hbase4:33441] regionserver.LeaseManager(133): Closed leases 2023-03-31 16:56:37,293 INFO [RS:0;jenkins-hbase4:33441] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-03-31 16:56:37,293 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-03-31 16:56:37,294 INFO [RS:0;jenkins-hbase4:33441] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:33441 2023-03-31 16:56:37,302 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,33441,1680281619375 2023-03-31 16:56:37,302 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-03-31 16:56:37,302 ERROR [Listener at localhost/44135-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@40b87682 rejected from java.util.concurrent.ThreadPoolExecutor@4487ea4d[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2666] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-31 16:56:37,303 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-03-31 16:56:37,303 ERROR [Listener at localhost/44135-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@334173dd rejected from java.util.concurrent.ThreadPoolExecutor@4487ea4d[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2666] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-31 16:56:37,304 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,33441,1680281619375] 2023-03-31 16:56:37,304 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,33441,1680281619375; numProcessing=1 2023-03-31 16:56:37,306 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,33441,1680281619375 already deleted, retry=false 2023-03-31 16:56:37,306 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,33441,1680281619375 expired; onlineServers=0 2023-03-31 16:56:37,306 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,43681,1680281617954' ***** 2023-03-31 16:56:37,306 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-03-31 16:56:37,306 DEBUG [M:0;jenkins-hbase4:43681] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@615ae3f8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-03-31 16:56:37,306 INFO [M:0;jenkins-hbase4:43681] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,43681,1680281617954 2023-03-31 16:56:37,306 INFO [M:0;jenkins-hbase4:43681] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,43681,1680281617954; all regions closed. 2023-03-31 16:56:37,306 DEBUG [M:0;jenkins-hbase4:43681] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-31 16:56:37,306 DEBUG [M:0;jenkins-hbase4:43681] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-03-31 16:56:37,306 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-03-31 16:56:37,306 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1680281620530] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1680281620530,5,FailOnTimeoutGroup] 2023-03-31 16:56:37,306 DEBUG [M:0;jenkins-hbase4:43681] cleaner.HFileCleaner(317): Stopping file delete threads 2023-03-31 16:56:37,306 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1680281620532] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1680281620532,5,FailOnTimeoutGroup] 2023-03-31 16:56:37,307 INFO [M:0;jenkins-hbase4:43681] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-03-31 16:56:37,307 INFO [M:0;jenkins-hbase4:43681] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-03-31 16:56:37,307 INFO [M:0;jenkins-hbase4:43681] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-03-31 16:56:37,307 DEBUG [M:0;jenkins-hbase4:43681] master.HMaster(1512): Stopping service threads 2023-03-31 16:56:37,307 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-03-31 16:56:37,307 INFO [M:0;jenkins-hbase4:43681] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-03-31 16:56:37,307 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-31 16:56:37,308 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-03-31 16:56:37,308 ERROR [M:0;jenkins-hbase4:43681] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-11,5,PEWorkerGroup] Thread[HFileArchiver-12,5,PEWorkerGroup] Thread[IPC Client (1760173019) connection to localhost/127.0.0.1:43983 from jenkins,5,PEWorkerGroup] Thread[HFileArchiver-13,5,PEWorkerGroup] 2023-03-31 16:56:37,309 INFO [M:0;jenkins-hbase4:43681] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-03-31 16:56:37,309 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-03-31 16:56:37,309 DEBUG [M:0;jenkins-hbase4:43681] zookeeper.ZKUtil(398): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-03-31 16:56:37,309 WARN [M:0;jenkins-hbase4:43681] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-03-31 16:56:37,309 INFO [M:0;jenkins-hbase4:43681] assignment.AssignmentManager(315): Stopping assignment manager 2023-03-31 16:56:37,309 INFO [M:0;jenkins-hbase4:43681] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-03-31 16:56:37,310 DEBUG [M:0;jenkins-hbase4:43681] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-03-31 16:56:37,310 INFO [M:0;jenkins-hbase4:43681] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-31 16:56:37,310 DEBUG [M:0;jenkins-hbase4:43681] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-31 16:56:37,310 DEBUG [M:0;jenkins-hbase4:43681] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-03-31 16:56:37,310 DEBUG [M:0;jenkins-hbase4:43681] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-31 16:56:37,310 INFO [M:0;jenkins-hbase4:43681] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=344.72 KB heapSize=393.13 KB 2023-03-31 16:56:37,321 INFO [M:0;jenkins-hbase4:43681] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=344.72 KB at sequenceid=574 (bloomFilter=true), to=hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/a76340964c234ca88b0f4a5063210714 2023-03-31 16:56:37,325 INFO [M:0;jenkins-hbase4:43681] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for a76340964c234ca88b0f4a5063210714 2023-03-31 16:56:37,326 DEBUG [M:0;jenkins-hbase4:43681] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/a76340964c234ca88b0f4a5063210714 as hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/a76340964c234ca88b0f4a5063210714 2023-03-31 16:56:37,329 INFO [M:0;jenkins-hbase4:43681] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for a76340964c234ca88b0f4a5063210714 2023-03-31 16:56:37,329 INFO [M:0;jenkins-hbase4:43681] regionserver.HStore(1080): Added hdfs://localhost:43983/user/jenkins/test-data/6f80227b-e8ee-98ff-f4dd-41e3d23f96a7/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/a76340964c234ca88b0f4a5063210714, entries=65, sequenceid=574, filesize=24.7 K 2023-03-31 16:56:37,329 INFO [M:0;jenkins-hbase4:43681] regionserver.HRegion(2948): Finished flush of dataSize ~344.72 KB/352989, heapSize ~393.12 KB/402552, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 19ms, sequenceid=574, compaction requested=false 2023-03-31 16:56:37,331 INFO [M:0;jenkins-hbase4:43681] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-31 16:56:37,331 DEBUG [M:0;jenkins-hbase4:43681] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-03-31 16:56:37,334 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-03-31 16:56:37,334 INFO [M:0;jenkins-hbase4:43681] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-03-31 16:56:37,335 INFO [M:0;jenkins-hbase4:43681] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:43681 2023-03-31 16:56:37,339 DEBUG [M:0;jenkins-hbase4:43681] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,43681,1680281617954 already deleted, retry=false 2023-03-31 16:56:37,405 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-03-31 16:56:37,405 INFO [RS:0;jenkins-hbase4:33441] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,33441,1680281619375; zookeeper connection closed. 2023-03-31 16:56:37,405 ERROR [Listener at localhost/44135-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4b604e97 rejected from java.util.concurrent.ThreadPoolExecutor@4487ea4d[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2666] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-31 16:56:37,406 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): regionserver:33441-0x1013b7eac850001, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-03-31 16:56:37,407 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@32497501] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@32497501 2023-03-31 16:56:37,407 ERROR [Listener at localhost/44135-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@594dcb85 rejected from java.util.concurrent.ThreadPoolExecutor@4487ea4d[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2666] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-31 16:56:37,407 INFO [Listener at localhost/44135] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 1 regionserver(s) complete 2023-03-31 16:56:37,505 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-03-31 16:56:37,505 INFO [M:0;jenkins-hbase4:43681] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,43681,1680281617954; zookeeper connection closed. 2023-03-31 16:56:37,505 ERROR [Listener at localhost/44135-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@448ec791 rejected from java.util.concurrent.ThreadPoolExecutor@6b73b90d[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4444] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-31 16:56:37,505 DEBUG [Listener at localhost/44135-EventThread] zookeeper.ZKWatcher(600): master:43681-0x1013b7eac850000, quorum=127.0.0.1:51916, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-03-31 16:56:37,505 ERROR [Listener at localhost/44135-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@343a90d0 rejected from java.util.concurrent.ThreadPoolExecutor@6b73b90d[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4444] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-31 16:56:37,507 WARN [Listener at localhost/44135] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-03-31 16:56:37,511 INFO [Listener at localhost/44135] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-03-31 16:56:37,627 WARN [BP-197372169-172.31.14.131-1680281614560 heartbeating to localhost/127.0.0.1:43983] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-03-31 16:56:37,628 WARN [BP-197372169-172.31.14.131-1680281614560 heartbeating to localhost/127.0.0.1:43983] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-197372169-172.31.14.131-1680281614560 (Datanode Uuid aeea1192-6f68-4306-8c37-723f39ed30f4) service to localhost/127.0.0.1:43983 2023-03-31 16:56:37,631 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/7dffd501-dbfb-c6fc-19df-043ab2e220e3/cluster_ab6d4253-9335-03d0-adb0-24d05d2df50a/dfs/data/data1/current/BP-197372169-172.31.14.131-1680281614560] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-03-31 16:56:37,631 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/7dffd501-dbfb-c6fc-19df-043ab2e220e3/cluster_ab6d4253-9335-03d0-adb0-24d05d2df50a/dfs/data/data2/current/BP-197372169-172.31.14.131-1680281614560] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-03-31 16:56:37,670 INFO [Listener at localhost/44135] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-03-31 16:56:37,857 INFO [Listener at localhost/44135] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-03-31 16:56:38,040 INFO [Listener at localhost/44135] hbase.HBaseTestingUtility(1293): Minicluster is down 2023-03-31 16:56:38,131 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(152): Removing adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint